James Blair: Part 2 — Solutions engineering, critical thinking, and staying human

Welcome to the test set. Here we talk with some of the brightest thinkers and tinkerers in statistical analysis, scientific computing, and machine learning, digging into what makes them tick, plus the insights, experiments, and OMG moments that shape the field.

This episode is part two of a conversation with James Blair, cycling boss, data junkie, and senior product manager at Posit.

From solutions engineering to product management

I'm so curious to get into your use of AI because I know you're a like preeminent AI tinkerer and doer. We've already talked about like bytes and filament. I think it's really clear that you have experimented with AI. I think for people to really appreciate the types of problems you're solving at Posit, I wonder if it'd be useful to talk just a tiny bit about what a solutions engineer is and sort of like what it looked like for you. Because my understanding is like you are solving a lot of interesting problems for solutions engineers now as a product manager.

Yeah, no, that makes sense. And I think some context here. So when I joined Posit, which was RStudio at the time, seven and a half years ago, I joined as a member of the solutions engineering team, which was tremendous. It was like this accidental perfect fit where I had been a long time user of RStudio and the software. And I attended the RStudio conference in San Diego in 2017. And I was just like blown away by the quality, not just the quality of the conference, but the community, the people, and particularly the people of the company, which I had never thought of RStudio as a company until that point in time. I had always just thought it's the software that I really love to use.

But then I put two and two together and was like, oh, there's people making the software. And I knew some of them, like I knew Hadley from his contributions to the space. And so I knew some of the names. But now all of a sudden I saw these people in real life. I had a chance to interact with them. And I was like, this is just, there's a tremendous culture. There's good vibes at RStudio. It was kind of the way that I left that. So I wanted, I knew from that moment, like I would love to work for this company and was just kind of like really trying to figure out how I would fit.

At the time, RStudio didn't have data scientists really working on internally. It was open source teams and then people building software. And I was not a software engineer and I was not like a prominent contributor to the open source ecosystem of our packages. So I was just kind of like, how do I, like, where's my entry point? And I found this listing for solutions engineer one day and the name engineer in the title or the word engineer in the title almost scared me away. I was like, I don't, engineer is not me, but I was desperate enough. I like clicked on the link and I read the job description and I was like, this is me. This is what, this is exactly what I like. I want to like, I know, I know RStudio software. I know that, you know, like I know the ecosystem and I'm really good at finding ways to like make things work in that intersection.

So, so I came in that way, came in on the solution engineering team, spent a number of years there, loved it. Absolutely. Like felt at home, felt fulfilled, felt everything that I wanted to and more from, from working for the company there. Do you mind recapping just for folks, like what a solution engineer is and does?

So there's two sides of this. There's like, what it was when I was there and then there's what it is now. So when I was there, it was very much like we, we worked with customers who find themselves in interesting predicaments. So we've got a great support staff at Posit, but then every now and again, there'd be issues where it's like, Hey, particularly when it was like, this is an intersection between they're using our commercial tools and they're also using some of the open source stuff. And they're trying to mesh these two things together in ways that we just are trying to make them successful at. And, and we would come in, solutions engineers would come in and kind of understand the problem, work with the customer and kind of ideate on solutions. And a lot of that, a lot of times what that meant is that we would get really interesting feedback that we could funnel back to open source development teams. We could funnel back to the commercial product teams because we were kind of like in the trenches with customers doing really interesting work.

And, and the fascinating thing is data's everywhere, right? So like you'd work with a significant pharmaceutical company one day, and then you'd be working with like a New Zealand dairy farm the next day to like solve very, very different problems, but with a very similar set of tools. And so it was really fascinating to just kind of think about how flexible open source is when it comes to like solving problems and, and really like, in my mind, re-emphasized the significance of the open source philosophy that we continue to, to kind of hold ourselves to as a company.

So that was, that was a lot of it. And then the other thing that was really beneficial was there's a lot, there was a lot of autonomy in the role at that time. So like you'd work on customer issues, but then sometimes there wasn't a fire to be put out. And so you just kind of had a chance to contribute to open source if you wanted to, to build out proof of concepts, write blog posts. So there was all this, you know, you kind of choose your own adventure. And again, for me, that just worked really well. I ended up working with a couple, I worked with the plumber package for quite a while and did some things there and really kind of dove into that world. I ended up working with a lot of the like partner groups that we have here at Posit which led to the role that I'm in today on the product side. So it just kind of found my way through various projects on the, on the solutions engineering side.

I didn't realize too, you were, you mean like you were opening PRs to plumber and you got a little code type.

Yeah. Working, working closely, like working closely with the team that managed that package, kind of figuring out, I made the, I made the plumber cheat sheet that still is in existence today, which was a ton of fun. And, and, and then ended up kind of as a, as a spinoff of that work, worked directly on the plumber Tableau package, which is like a interface between plumber APIs and Tableau extensions, which was, which was a lot of fun and kind of very similar to the partner work that I'm still engaged in today.

Yeah. I feel like the thing is too with cheat sheets, it's like people love cheat sheets. I feel like nothing hits as hard as a good cheat sheet. Uh, somehow does it, our studio has like this whole repo of cheat sheets that like.

Yeah, there's a whole bunch of them.

If they went away, people I think would riot and burn the internet down. It was also one of the times I remember working on that. I have a, I have a giant, like my monitor's a 37 inch behemoth and, and I use all, normally I have it split in like four panels as, as if it's four separate monitors, but I don't know, it's 43 inches. It's huge. Anyway, I remember working on the cheat sheet. I use like all screen real estate and I just had like, I think I built it in, I think I designed it. It made it, it might've been keynote. Like it was kind of like a design software choice, a little bit odd, but I just remember like just sitting there at my desk for days on end, like building out this cheat sheet and painstaking detail. And I'm so glad I have this giant monitor. Cause I could like step back from my desk and like see the whole thing and then like get back into whatever part of it I was putting together. It was, it was a fun time.

And you have to think critically about like what, what should be included? Like what is, what is really important as a reference point? What's not important. And people can kind of look at other documentation sources for us. So yeah, that was, that was a, that was a fun project.

AI-powered demo generation

I want to get back, you know, to the point that you brought up originally, which is on the AI side and the work that I'm doing now. So coming from the solutions engineering side, it's interesting, like at Posit, a lot of the product management team is made up of former solutions engineers, just kind of the path. Is there a reason for that? Do you think?

I think part of it is there's something to be said about the solutions engineering really helps you learn how to listen to customers and users, right? Like I keep using the term customers, but we can, this extends beyond just like people paying for software. This is people using software. And with our commitment to open source, there's a lot of users that are equally as important to us. And so really learning to listen to feedback, understand pain points, understand use cases, and then be able to kind of step back from that, look at the bigger picture and figure out ways that we might be able to solve for those things. I think you really become good at that in solutions engineering. And that's a really critical skill for product management as well.

So I think that like, it just kind of became this path that developed on the Posit side. And the product team has been built up over time. Like we didn't really have a formal product team when I joined, and that has since changed where we do now have a well organized, well built out team of product managers. And so I think just kind of like over time people as they've had the opportunity and the desire have made that migration over to the product team.

So I think there's, you know, the AI saturated world that we live in is, I think, challenging to navigate in the best of circumstances for a number of reasons. But there's also, in spite of all of the like noise and chaos that I think exists right now, there's also quite a bit of excitement. If you kind of know how to look for it, I think there's, you know, it's an incredible time to kind of like build and think about solving problems because it's really easy to build proof of concepts when before it would take a while. And so there's just like these, there's ways you can kind of operate in this world that I think are really unique.

And so one of the things, just to give a glimpse into some of the stuff that I've been involved in recently, one of the things that we've kind of, that I've thought about for quite a while, particularly given the background in solution engineering, is this idea of we're constantly kind of trying to tell the story of what our software does. And so we're often sitting in front of a Zoom call or in person with an organization, trying to kind of help them catch the vision of what sorts of problems we can solve. And traditionally, we're really good at telling that story with like empty cars data. It's like, hey, here's some like, here's some pretty simplistic data, but let's just like, if you can abstract this enough, you can kind of see how this could apply to you, whoever you are. And that puts a lot of burden on the learner to kind of like, build the right mental bridge between what they're looking at with, you know, the difference between four and six cylinder cars and their miles per gallon, and like the genomic data they work with day to day, right? Like that's a big leap to make, but we ask them to make it.

And so I've always had this like, man, I wish we could be better at delivering like meaningful, engaging demos and walkthroughs and content, but it takes a lot of time to like, build that stuff. And I, you know, like we've a solution engineering that was one of our primary things was building content for demos and blog posts. And what would happen is we build something, we invest a lot of time, a lot of cycles into this thing. And then that would become the new thing that we reused over and over again. Like for a long time, I think this still exists. We had like this bike share example that used open bike share data from Washington DC. And so we had like a shiny app built around it. We had a machine learning model that was trained to predict the availability of bikes at certain stops. And so, and then all of a sudden, like every customer got to hear about Washington DC bike shares, because like it was the, it was the example to show, but they weren't a bike share company. So they still had to make the mental bridge between like, what is that? And what is the data they actually work with?

So a few months ago, we were, I was having a discussion with somebody internally. And I was like, you know, well, like, I, I feel like the tools and the things are there for us to be better at this, to be able to kind of build targeted content more efficiently than, than what we used to do to the point where maybe we could start being a lot more targeted with the conversations that we have and the data that we show and all these pieces. So I talked to my boss about it, said, Hey, like, here's this, you know, I'd love for us to do this. And his response was something along the lines of we've, we've tried that. And like, he like, yes, that's a great idea. But how do you find the data? Like, how do you find industry relevant data? Kaggle, right? There's, there's certain sources that people typically will turn to open, open GitHub, whatever, right? Like you can find data, but I know that game. And I know that I'll be like, yeah, let me go find data. And like six hours later, I'm like, there's no good data here. Like, it's just, it's, it's really hard to find good open data.

And so I was like, I was, it was on the road at the time. It was actually in San Francisco, but I remember I was in the hotel room that night and I was like, okay, like, how do we, how do we find the data? Oh, I was using Positron. So I had Positron Assistant up and running. And I was like, Hey, Positron Assistant, can you, can you make me some data for this industry? And, and then it did it. And the, and the, and the key thing about this was it didn't just like spit out a bunch of data. It wrote a bunch of code to generate the synthetic data. So now all of a sudden, like, I'm not bound by what the LLM can like regurgitate. I'm letting it write some code that writes some synthetic data that can actually be surprisingly high quality. It's synthetic. And like, there's no getting around that. And I'm, I'm actually like, I think that's fine. I think it's fine to be like, this is some made up data, but it's really cool to say, here's some made up data that is really potentially indicative or familiar to you versus here's some information about bike share usage inside of Washington, in Washington, DC.

I think it's fine to be like, this is some made up data, but it's really cool to say, here's some made up data that is really potentially indicative or familiar to you versus here's some information about bike share usage inside of Washington, in Washington, DC.

And so there's this like shift where I was like, Oh my gosh, like we can use AI to help us like make data up that can be really illustrative and really powerful. And so then we built out this sort of whole internal tool chain that basically says, Hey, like help me make some data. And then on top of that data, help me write some, like a Quarto document and a shiny app and build a NL pipeline. And like, that way I can show some of these things off with data that's so much closer to the reality for the person I'm talking to versus some of these like more static assets that we've used in the past. So I've been really excited about that. It's like, we've had, I've had a ton of fun kind of like exploring and building that out. We've leaned heavily on Claude Code to kind of help us build out what we, what we now have today. In fact, I just, we had an update to this tool yesterday or two days ago where like Anthropic just announced agent skills as a new thing that you can manage inside of Claude Code. So this is now available as like an agent skill that you can just enable inside of Claude Code. And then you can be like, Hey, write me a demo for this industry. And it goes and makes a bunch of data and then writes a book, you know, writes a shiny app or a streamlined app or whatever you want. And, and then you've got this like fully encapsulated little project that you can show off and it takes 30 minutes of Claude just kind of doing its thing to get there. It's like phenomenal.

Like today I can talk about this, like, of course this happens. But when I first like started building this out, it was repeatedly just me with my jaw on the floor being like, I had no idea that like we were at this point with AI where this kind of thing can happen so quickly. Because I know that if I were to try to do this myself, it would be like two weeks of work to get all this together the right way. And now it's like an afternoon of me telling Claude what to do while I go eat lunch and I come back and I have something really cool.

Yeah. Let the, you went from, it sounds like from trying to find the one perfect example to creating the like example generating process that can just fire up like great examples kind of on the fly.

The thing that's been the best about this, in my opinion, is there's kind of this pattern that's emerged where we've got, you know, we have Positron Assistant that we've put out there. We also have DataBot as this sort of exploratory data agent that, that I, I love, I think is a, is a fabulously designed, well-built tool. And so one of the things that's been really fun is to kind of like join these conversations with users who are coming from all kinds of backgrounds there, you know, they have their domain expertise and we say, Hey, like here's some synthetic data that is perhaps illustrative of the kind of data you work with day to day. And, you know, here's what, here's what that can, here's what it contains. Again, it's all made up, but this, this can help us have a good conversation. And then we pull open DataBot and we're like, Hey, let's explore this data together. And I'm sharing my screen or, you know, showing them my laptop and I'm explaining this data and I give it a couple of prompts. So we look at some things and then it's really, it's been so fun to ask, what would you ask this data? Like, what would you want to know?

And then they respond with, Oh, I, I often look at this or I'm really interested in, like, I'd be interested in understanding what, how, like how these two things compare. And then you let them kind of participate in the process. And all of a sudden, this goes back to what I talked about earlier, like my level of teaching, right? Like you can just see the light bulb go on where they're like, Oh my God, like this, this is what my day to day could be like, I could just be like chatting with this thing that's helping me explore the data. And then it gives me the like cognitive freedom to think more critically about the types of questions I should be asking here. Like, what is it like instead of me worrying too much about like writing the code and remembering the exact syntax for which ggplot theme I want to invoke here, I instead can like think a little bit more deeply about how I can apply my understanding of this domain to the particular data that I'm working with right now.

Daily AI tools and workflows

It's interesting to hear. It's like you have an entourage, if I understand of like bots, you've got this thing, which I think it's called demo bot. If I'm allowed to say your internal tool, then you've got like DataBot, which is the Positron. What are the Positron? Like data analysis assistants. You've got like a bot posse kind of like in tow for different parts of the process. I wonder if it'd be interesting to talk a little bit about like what tools are using day to day to interact with AI. So it sounds like you're using Claude Code a bit for your demo creation. Maybe, maybe you could give us a little bit of context on like, yeah, which, which tools are using, and then we can go around and maybe talk a bit about which bots, which tools we're kind of working most with right now.

I have an interesting, maybe it's not interesting, but like my philosophy there is I, I think there's high value in exposing myself to a lot of tools. So I have lots of things that I kind of like rotate and cycle through. So I, I, I really enjoy Claude Code. I like to use that as kind of like my quote unquote daily driver for sort of agentic style work. And then, and then their, their ecosystem and traffic continues to do all kinds of really interesting things with, with what Claude Code is capable of. And so there's all kinds of like interesting community contributions that I'm constantly kind of like tinkering with and exploring. I, I use Codex on the open AI side as just like, Hey, how does this compare? What does this feel like in comparison to some of the other tools I know?

I'm a big fan. Wes pointed this or mentioned this in an internal Slack thing a while ago, but there's a tool called Conductor that I really like that, that is essentially like a wrapper around Claude Code, but it is really well-designed in terms of it lets you just run multiple Claude Code sessions against different work trees for a given project. So you have these like independent agents running on their own clean copy of your code and they can all be running on like totally different tasks without the risk of interfering with one another. And then once something has reached a point where you want to kind of absorb those changes back to everything else, you can submit a PR and like, like just the, the, the mechanics of it is really nice for like a multi-agent sort of workflow. I've, I've found that to be a, a really, a really nice tool to use and, and one that I kind of continually go back to.

Oh, wow. Yeah, that's a wild one. Yeah. Wes, what are you using these days?

Uh, I mean, like James, I mean, I've, um, for the last maybe six or seven months, like I've, I've, uh, been solidly using Claude Code as my quote unquote daily driver for doing most of like mundane work, building bespoke tools. Like I feel like, I feel like there's this fun analogy between 3d printing, like building an exact widget that solves a problem that you need and using, using a coding agent to build the exact, uh, script or special specialty tool that solves, solves the problem in exactly the way that, that you want or like automating. So I've been going through and, you know, yeah, building all kinds of stuff and building bespoke tools to automate, you know, my, my day-to-day day-to-day work, because I'm, I think in my open source work, like I abhor drudgery and inefficiencies. And so now like having the ability to use, uh, agentic coding tools like Claude Code to build, um, essentially to, to dive into types of development that I would never have touched before. Like I've built recently, I've built terminal UIs, like, you know, I've done lots of work on CSS. Like I I'm improving like all of my websites, like making CSS improvements, design improvements that I never would have touched before. Like I would have needed to consult a designer or somebody because I'm just not good at that type of work.

So, you know, I've, I've been treating AI really as a tool for leverage and for delegating, uh, work that, that I find to be low value for me and, and to help free me up to, to spend more of my time thinking about some of the higher level concerns. But the big, big learning I, I have from the last six or seven months is, is that these tools really only work as well as you can articulate yourself and, and know what to ask for. And so if you aren't articulating clearly and in great detail, what you need, the, the, the agent to create for you. And then if you aren't looking at the code that it's generating and scrutinizing it and actually giving a good quality feedback about what it's doing wrong or what it's not doing, uh, it can really create a gigantic mess and that can be a huge, that can be a huge liability.

And so I, the way, what I've been telling everyone the last six months or so is that, that there's this going to be this increased urgency for everyone to be much more, much better at code literacy and, and reading, reading code and, and to, uh, spend more time doing like analysis of code that they didn't write. And to be able to like read and give, give feedback on code because we're all going to be writing a great deal less code than we did in the past.

There's this going to be this increased urgency for everyone to be much more, much better at code literacy and, and reading, reading code and, and to, uh, spend more time doing like analysis of code that they didn't write. And to be able to like read and give, give feedback on code because we're all going to be writing a great deal less code than we did in the past.

But, uh, I know, yeah, I'm, I, I should, should broaden my horizons and learn more tools. But as long as I'm, I'm, uh, long as I'm productive using Claude Code, that's, it's been sufficient for my needs in recent times.

Yeah. It's super interesting. I definitely think on the theme of like picking up new tools to like, that's something I struggle with. And I have to admit, like, I really want to get into Claude Code, but haven't yet. And I'm like, so ashamed to say that out loud. I mean, you're missing, you're missing out. That's all I can say. Do you have tips? Like how would you suggest someone like start dips their toe into Claude Code?

Try building something totally greenfield. Like look at your, it could even be something that's not work related. Look at your personal life. Like I, you know, really any, really anything just, and to, to sit down and build something, especially something that you would never have, have built yourself. Like I've heard of people building like Android applications, people with zero Android application development experience. And like it, it, the ability to like break through and tackle problems in spaces where, um, you have, have little experience or maybe not little experience, but maybe little interest in developing expertise. I think that's, that's really, that's really interesting, um, in building production code for enterprise software that, that also comes with a downside that, that if you step outside of your lane where you're unable to give good quality code review and feedback on code, maybe I would be less, you know, maybe a little bit less enthusiastic.

I also find it's, it's as a tool to like do, you know, uh, refactoring and augmentations of existing unit test suites. Um, like you can say, Hey, look for code duplication in this file, create some helper functions, refactor things, make things clean, normalize doc strings. Like this really, the sky's the, the sky's the limit. And it's all really, it's all limited by your ability to identify things that you want to do and ask the right questions.

Yeah. Yeah. It's interesting to hear too, like the Android app, uh, like, um, like the difference between like what you could imagine yourself doing and like, if, if Claude Code could build an Android app, it would really freak me out because I can't, there's like no world where I see myself doing that. So maybe that's like really freaking myself out with Claude Code.

Yeah. I think it's, I think it's fascinating. I think like the Greenfield example is a really good way to dip your toes in the water because it just like, it's low stakes. And honestly, it's just a matter of being like, here's, you know, like this, the easiest way to start is to describe as clearly as you can, what you're trying to build to the model. And, and then like learn, there's all kinds of, you know, like there's roughly 8 billion blog posts about how to use these tools correctly. And some are valuable and some aren't. But I think that the biggest thing going back to an earlier point is to just like put hands on and experiment and find something that is of interest.

I've, I, an example from my own experience has been like, I'm not, I'm not a web developer by any stretch of the imagination. And it's something I could, I could certainly spend the time to get better at and get good at. But I've had, I've got a few different websites that I've built out that I've just used Claude Code and kind of like vibed with. And, and it's, to Wes's point, in the process of doing that, I feel like my ability to kind of understand TypeScript and CSS and some of these other things has elevated as I'm like reviewing, do I understand like everything that's written here? No. Could I reproduce it myself? No. But I know more now than I did when I started this project and continue to kind of iterate.

I think there's a, again, like there's, there's kind of this intentional effort to say like, I'm going to, I'm not just going to like tune out and let this thing do the work for me. I'm going to stay an engaged participant. And that, what that looks like is kind of what Wes described, I think like I, my role is now as a reviewer and as a tester and as somebody who needs to communicate back to this model, what is not right or what, you know, like, and it's been really like, again, just that Greenfield example, I think you can end up in a place where you find where these things are really great. And there's areas where there's just the capabilities of these tools are phenomenal. And then you'll find yourself in something that feels so similar. And all of a sudden the model is so bad at it. And you're like, but you were so good at this. And you just realized that, like, again, there's this, this like very jagged abilities curve to what these things can do. And so it's like, just through experience, you start to kind of understand, okay, like, here's where these tools struggle. And I have to kind of scaffold them this way to make it work right. And then these other areas, they're a little bit better to just do their thing.

I think my, I think my advice would be different to getting started. I would say I would instead like find an existing project where there's like something you've always wanted to do for a long time, but you keep putting off because every time you look at it, you're like, oh, this is going to be like, way too hard. And just like throw a cold coat at it. And there's like, maybe a 20% chance, like it does the right thing or like does it, answers it well. But even if it doesn't do a good job, like just that seeing someone's tackled this, this problem that in your head is like really complicated and seeing that, like, maybe it's got 80% of the way there with like a way less effort or, or like it's done something obviously wrong. And then you want to like step in and correct it. It's just like such a, I think it's like a really great way to get like unblocked when there's like some tasks that you're just kind of like stuck on psychologically. So that, that's to me where I found it like the, the most valuable, like certainly I use it for all the stuff and like lots of like little routine changes and like great for refactoring or when you're trying to do basically the same thing in a bunch of different places, but that's kind of like unlocking I've found to be like super, duper valuable.

Do you think, are there any nice aspects of the Tidyverse that are thanks to AI taking something that you dreaded doing or felt stuck on and blasting it?

Not the Tidyverse yet, but I have been, I did a bunch of work on TS that, um, like there were, there were issues that had been sitting in TS that for like 10 years, not necessarily because they're like so incredibly complicated, but every time I'd look at them, I'd just be like, oh, this is going to be really hard and the payoffs not that big. So I just like kick the can down the road. And I keep doing that for like multiple years. And so just having it, have a go at some of those things in my experience was maybe like a third of the time it would come up with us. So like, I was like, oh, this is a really easy solution. Um, and it looks great, but like a third of those times I would later discover that it's actually solution was like actually too simplistic and it didn't really fix it correctly, but it's still kind of like got me over the hump. And then I had to do like more work, but at least it kind of got me unblocked. And then other times it's just like sometimes having an obviously wrong answer in front of you is more compelling than having no answer at all, because you're like, okay, this is wrong and then fix this.

Uh, oh, I was going to ask, are you using Claude Code? What kind of things are you? Yeah. Nice. Yeah. Okay. I'm the odd one out.

Yeah. It's really, I think the thing that's great about Claude Code is you're just doing it in a terminal. Like it doesn't require any like special integration with your editor. Um, I do, I do all, I also sort of wonder that this like, you know, Claude Code has this like very, I don't know, like kind of 1990s, like user interface. And I think there's like something like about that nostalgia that is like comforting to engineers who are like a little bit worried about AI, like taking over their jobs. It's interesting. I would let AI take over my job if it felt like Neuromancer, you know, like maybe that's, that's the key. Yeah. The problem is like right now the reality seems to be more like, I'll take over your job by producing a ton of stuff, but then you have to like carefully go through a line by line. Yeah. That's fair. Giving me a new job.

I also wonder, like, I remember like, you know, years and years ago, like I've always like, I think like many programmers, like really into keyboard shortcuts instead of using the mouse. And there's like this kind of scorn for like, why would you use the mouse? It's like so much slower. But I remember reading an article that actually keyboard shortcuts aren't actually that much faster. They just like cognitively feel faster. And it does feel a little bit like this with many of these AI tools. Like it feels like you're moving much faster, but are you actually moving that much faster? Or like there's certainly your speed is much greater, but is your like velocity greater? I don't know, but absolutely. Well, like I had to believe every, every day a scientist, every software engineer needs to be engaging with these tools and like trying them out and learning like so, so much of what you read is just like hype or abject skepticism. Like you have to try it out yourself and, and learn.

I think too, from a, from a product management perspective, it's been really interesting because it's, there's a lot of discourse right now in the product management space of like, what does this mean for the future of product management? Do we see this like convergence of engineering and product work where like product people that are the builders and vice versa kind of thing. And who knows like how that's all going to shake out and materialize. I do know that it's made certain things like prototyping and like putting proof of concepts together a lot more accessible to, to like the product side, which is really nice.

And then the other thing that I've seen in the practice that I've started kind of adopting is every now and again, I'll find rough edges from customer conversations or whatever it is, right? Like I find something that I'm like, Oh, like this should, it would be nice for this to be changed, or it would be nice to have this feature. And if it's, if it's approachable enough that I feel confident in my ability to define the problem. And if it's in a part of the ecosystem that I feel like I can operate safely within, I'll take a, like, I'll take a first pass at it with myself and Claude Code and, and, and build out something and, and then make sure that I have a full understanding of what I've built and have kind of gone through the code review process myself, and then pass it back to whoever the maintainer or, you know, whoever the owner is from an engineering standpoint and say, Hey, like, here's this issue that I've filed. I've also created this pull request in response to it. Here's, I understand what's going on. I've tested this, but you happy to chat about it, but it kind of helps me meet the, the, the engineers kind of where they are.

What I've had to do though, very specifically is avoid the kind of natural inclination to just like vibe something out and then toss it into a PR and be like, hope this works because then you're just like shifting cognitive load to somebody else, right? Like, I don't, this shouldn't, this shouldn't be convenient for me and a burden for someone else. It should feel like it was a convenience for both of us or however many people are involved, right? Like it should feel like we all won versus like, Hey, I got a like shortcut, but my shortcut was actually like making you do more work. Like that's not, I don't, I don't think that's the right way to, to use this. And so I'd been really conscious of saying like, I, if I'm going to put my sort of engineer hat on for a minute and use some of these tools to try to pull a solution together for a problem that I, that I feel passionate about, then I want to make sure that I can defend what I've submitted and the solution that I've proposed because I understand it. And, and that is the kind of critical piece for me is like, I don't, if somebody, if an engineer wants a vibed solution to their problem, it's really easy to just like wire up Claude Code through GitHub actions and tag it to go solve an issue and give you a draft. Like they can self-select into that sort of feedback. What I can bring is like, as a human, I reviewed this, I could talk about it and we can reason together about if this is good enough or if we need to do more.

Claude skills

Yeah. Yeah. Keeping the human in the loop a little bit. It sounds like, I want to really quick, I know we're running up on time, but I was hoping really fast just to go around. Cause I know Claude skills are relatively new and there's been a lot of chat about Claude skills. Maybe we could just go around and for like 30 seconds, a person just quick, get people's impressions of Claude skills. Cause I know there's been a lot of like internal discussion and things like that. Wes, what do you, what do you think?

We've been using, I mean, we've been using a, it's just like prompts, like markdown documents in the dot Claude folder to teach Claude Code, how to do stuff, develop and workflows and help with different specific areas of Positron. And so I think what's cool about Claude skills is it's a more structured approach to something that we were solving in a more, a more ad hoc way. And so I see it as something as like Anthropic responding to something that's a genuine need, especially in more complex projects where, you know, your project has different ways of working and there's complexities involved with development workflows and how you also interact with the project, how you create issues, things like issues in GitHub and things like that. So I haven't used it, but I have like a pull request. There's already a pull request up in Positron to port some of our existing Claude Code machinery to use the new Claude skills stuff. So I will be looking at the head and getting more familiar with it since I haven't, I haven't used it myself yet since it's just, it's only a few days old.

Yeah. It sounds like it's kind of a natural interface across this pattern of having markdown files to explain things. Hadley, what do you think?

Yeah, I haven't used it yet either. I'm excited about just adding a bunch of skills for kind of bits of like, like particularly kind of recent changes to the way we develop like packages and aft. So for example, like recently I was using Claude Code to generate some tests and I wanted to use mocking and I had like no idea how to do mocking or test that because it's a relatively new feature. So I just, that to me, the potential of skills to kind of be like, okay, you need to use this. Like, you're not going to, you don't, you don't know about this. Here's the, here's the, you know, quick, here's a quick briefer on like how you use mocking or test that or how, here's how you use snapshot test. So here's how you use F7 just so if I'm going to use a really funny features, Claude knows how to use them.

Oh, awesome. Let's see, James.

Yeah, it's, it's rare that I feel like I'm ahead of the curve because I have used skills and I think it's, to me, there's a, there's a lot of upside here related to a couple of things. One is like portability, right? Like skills can be really portable and, and shared really easily. And I know that like the dot Claude folder is part of that, but also like with combined with Anthropic's new plugin interface within Claude, like you can, you can easily just like install new plugins and that can include new skills. And then the other thing, the thing that I think is probably most significant from my perspective is that skills operate in this way that Anthropic describes as progressive disclosure, where when Claude starts up, it only familiarizes itself at the highest possible level with the description of what each skill does. And only when it determines a skill needs to be invoked, does it load the rest of the information from that skill into context. So you can have like this huge catalog of skills that takes up a very small fraction of your context. So your context stays really clear. And then, and then you ask some requests and Claude says, oh, like there's this skill that I should use for this thing. And then it invokes that skill. And then even there, there's like continual progressive disclosure where it will like read the markdown file or files associated with that skill.

And if certain facts like, like, so let me give you an example. Sorry, this is longer than you anticipated, but in Demobot, you could choose to use either like Python or R for a demo project. And there's instructions for each one, like how to build a Python project and how to build an R project. So if you ask Claude Code, Hey, like build me a project and use R, it won't even read the Python documentation. Like it knows that that is not part of what it needs to know and understand for this particular task. So it keeps, it keeps that out of context and just reads the bits that are relevant to the specific ask. So I think like that's that part of it, the portability, and then like this ability to like define lots of skills without clouding context is, is going to be, is going to be a big deal.

Staying human in data science

And I guess sort of white down any last advice for people getting into data science today, you give, I think it's just an echo of kind of what I've already said. Like the fact this sounds maybe stupid, but like the fact that you're human is like your greatest strength, you know, like the humanist that you bring to data science is the value. The value that we produced was never the code that we could write in my opinion, right? Like it was the questions that we asked and the way that we thought about data. I, I look at the, my background and history and everything. And one of the great sort of gifts that I think data science has given me in the pursuit of this field as a career was the ability to just like reason about and think about data. And that has nothing to do with programming necessarily. It's just like, how do I map data mentally to think about like how to, like what questions can I ask? How do I structure those questions? And then eventually that translates itself to code, but like to Wes's point, when we were talking about AI, your ability to clearly articulate what you're trying to accomplish to the model is really critical. And that has very little to do with code. It's just, it's just communication.

So it's about like, how will I understand this data and using my curiosity and the things that make me a human, my critical thinking capabilities, all these things, I can then like describe what I'm trying to understand and let a model kind of write the code for me because the code that's written was never really the point. It's like the outcome that we're driving to the understanding that we're trying to get to, like, what is, what is this collection of information telling me and what do I do as a result? And like, that's what we're trying to get to. And, and it, we have to, we have to stay human to do that the right way. So that, that's my advice, like remain curious, stay human, like use these tools to help you be better at what you do best, be better at being curious, be better about like asking the hard questions, be very, be better at like critical thinking, because now we have an opportunity to, we have these tools that kind of take care of some of the other stuff for us. Don't use that as an escape valve to, to be lazy. Use that as an opportunity to be better at the things you're uniquely suited to do.

Remain curious, stay human, like use these tools to help you be better at what you do best, be better at being curious, be better about like asking the hard questions, be very, be better at like critical thinking, because now we have an opportunity to, we have these tools that kind of take care of some of the other stuff for us. Don't use that as an escape valve to, to be lazy. Use that as an opportunity to be better at the things you're uniquely suited to do.

Yeah, James, really appreciate it. I think hearing about your path from like robotics to journalism to data, and especially with things like product management and that whole sort of path is really interesting to hear. And I think when you wrap it all up with all your tinkering and your work with AI, it's, it's interesting to hear how all of it's kind of come together and that you're very much like the human in the loop, doing a lot of, a lot of this interesting data work. So really appreciate you coming on to The Test Set and excited to see you, all the stuff you do. Thanks for having me. Thanks, James. Thank you.

The Test Set is a production of Posit PBC, an open source and enterprise tooling data science software company. This episode was produced in collaboration with creative studio Agi. For more episodes, visit the test set.co or find us on your favorite podcast platform.