Data Science Hangout | David Meza, NASA | People analytics for getting to the moon

Hi, everybody. Welcome back to the data science Hangout. If you're joining for the first time, I'm Rachel and it's great to meet you. So if this is your first Hangout, what is it? Well, this is an open space for the whole data science community to connect and chat about data science leadership, questions you're facing and what's going on in the world of data science. So these sessions are recorded and shared to YouTube, as well as the data science Hangout site. So you can always go back and rewatch or find helpful resources too.

I'll say this up front as well, we do have a LinkedIn group, too, that people can join and I'll have Tyler share. Oh, she just shared it right there in the chat right now. If you ever want to continue a certain discussion or connect with people that can be helpful too. We always want to create spaces where everybody can participate and we can hear from everyone. So there's three ways you can ask questions today. As you may know, this is all audience led. So you can jump in by raising your hand on zoom. You can put questions in the zoom chat and just put a little star if you wanted me to read it out loud instead, or I could call on you. And then Tyler will also share a link to our Slido where you could ask questions anonymously as well. Just like to reiterate, we love to hear from everybody no matter your industry or level of experience as well.

But today I'm so happy to be joined by my co host, David Meza. Your title is a mouthful here. So AIML R&D lead of people analytics at NASA. And David, I'd love to kind of turn it over to you to share a little bit about your role and the work that you do.

Great. Thank you, Rachel. Appreciate the introduction. So hi, everybody. Glad to be here taking this hour with you guys. So my role currently at NASA. So I joined the office of the chief human capital officer at NASA headquarters about four years ago. My career at NASA is a little over 20 years, most of it down at Johnson Space Center as the chief knowledge architect.

But what I'm doing currently right now, it's very similar to what I did at JSC on the knowledge architecture, but it's basically taking a look at our infrastructure in how we take data from its authoritative source and turn it into some kind of actionable knowledge. So I look at that from a framework of what I call again, what I call knowledge architecture, which is a combination of knowledge management, informatics and data science, where we look at knowledge management as the strategy for how we collect, we analyze, we store, we present and visualize our data. Informatics is the pipeline for transforming or transmitting that data from the authoritative source to the presentation layer. And then data science are the methods and algorithms we use to turn that data into some kind of actionable thing.

So in people analytics, I'm applying those concepts within human capital to see how we can make a very cohesive workflow between all of our four architectures. And I define the architectures as the data architecture, the IT architecture, the analytics architecture and the software architecture to make sure that we can get the data, we can integrate it through the IT, we could analyze it and eventually present it to our individuals. We've got a long way to go within people analytics to try to modernize our stack, to see how we can do things. So it's kind of my job to make sure that all parties are talking to each other, we're thinking about things, we're including the right tools such as RStudio, which are, you know, and different types of things that we're trying to utilize it and help our data scientists to answer the questions that our end users are having. So that's kind of a brief rundown of what I'm doing right now.

Excitement about data science and people analytics

That's great. Thanks, David. But David, I'd love to ask you what's something you're most excited about in the next year ahead with regards to data science?

Something I'm most excited with regards to data science. I think it's well, it's one of the reasons I came into this into people analytics is because I really felt that data science was becoming, was really starting to become mainstream within people analytics. So I'm really excited about the aspect of how we're utilizing various concepts and technologies. A lot of my focus is around natural language processing, as well as graph algorithms and graph databases. So it's the combination of actually being able to utilize the graph databases and graph algorithms programmatically in connecting to our data and creating those pipelines of finding information within our text, creating that knowledge graph, and then being able to apply various algorithms to it to deliver answers.

And just a simple example of what we're doing there is around our skill and competency capabilities. As you know, we're trying to get back to the moon and onto Mars, but we need to identify the various skills and competencies that we have in our individuals. And just from a human perspective, that's very labor intensive. So we're trying to use some of our, some of these concepts that I've just mentioned to be able to extract NASA specific competencies from individuals or people that may be applying for jobs based on some textual reading and then creating a graph of those competencies to see how they align to our workforce. And they're starting to develop pipelines and methods to say, we've got gaps here, or here's your career path, here's your training path. Here are the number of individuals that we have that meet 70% of our skills, and we need to upscale them to the next 30%. And we've identified that through various algorithms. I mean, that's a, that's a lot of words for just basically saying, you know, we're just trying to make sure that our staff is appropriately aligned to get to the moon and onto Mars.

A lot of my focus is around natural language processing, as well as graph algorithms and graph databases. So it's the combination of actually being able to utilize the graph databases and graph algorithms programmatically in connecting to our data and creating those pipelines of finding information within our text, creating that knowledge graph, and then being able to apply various algorithms to it to deliver answers.

And I would love to hear what are some of the NASA specific competencies?

Well, I mean, when you look at it, I mean, a lot of them are, are pretty close or pretty aligned with, with engineering, of course, electrical engineering, mechanical engineering, different types of research. But when we're talking about getting onto the moon, you know, there's, there's propulsion capabilities, aerospace type capabilities that very few people have done. And things that we've done that we've, unfortunately, we've forgotten about, you know, there's, we've had to go back to our Apollo era days and look at, you know, how we, how we use the parachutes to land the rocket, the capsule back into the ocean. We forgot how to do that. So we had to go back and test that out again and redo it. It's because of our knowledge management capabilities. We, while the information was stored, we didn't keep it in a way that it was easily accessible. So we've learned from that. But definitely propulsion is a big issue. Robotics and how we utilize robots right now on Mars with, with Robonaut on the Mars lander, things like that, that take some specific skills and understanding of how to get those mechanisms onto a, another planet or onto the moon.

How David got to NASA

That's awesome. So there are a few questions coming in on Slido right now. And one of the first ones was, David, NASA is like a dream company for lots of people and kids. How did you get there?

How did I get here? You're testing my memory here. Well, actually it was luck maybe. Fortunately, I grew up in Houston and I grew up in the area where Johnson Space Center was. So I knew a lot of individuals that had worked for NASA or were through there. I was actually in industry. I was running a small company and really decided to get back into computers. I mean, this is a, this is the day and age where computers were just starting. I started off as an engineer and went into computer science primarily because the computer science majors were doing their coding on a monochrome monitor system, an 8088 computer rather than punch cards because I didn't want to do punch cards within the engineering department. So it was just knowing people at that time and applying for a job as IT was really booming. They needed IT help. I was able to get land in an IT position with a contractor of NASA down at Johnson Space Center. And after about six years as a contractor, I was able to then apply for a civil servant position, which I was hired on upon for that. So it's really just keeping an eye out for the opportunities that are there. Keeping your resume up to date. And then it definitely does help if you have somebody that can provide a reference or understanding of how NASA works. It is a little competitive. My last announcement that went out, I probably had close to three or 4,000 applications that came in. So it can be competitive, but it's not undoable.

Analytics in recruiting and performance management

So this might be, this question is tied to that then. Someone asked, how baked into recruiting and performance management are the analytics you're working on? Is this more aspiration or practical currently?

Can I say semi-practical? In a sense, we've got some good performance capabilities. We've got some good recruiting capabilities that are out there that are probably a little more old school, utilizing older technologies or utilizing some manpower. The analytic components of it are more aspirational right now. I mean, we do have some decent attrition models. We do have some time to hire models that we're utilizing that are in production. But a lot of the other things we're looking at are still somewhat aspirational, but we're moving a lot closer. Kind of why they brought me in a couple of years ago was to get to that point where it's more production ready. Think about what we need to do, implement the architecture, work with the agency to implement the architecture so that we can do those things analytically that we're trying to do. So it's a work in progress, but like anything in the government, unfortunately, we run a little slow. So it takes some time to get these things developed.

Team size and structure

Thank you. Somebody else had asked on Slido, how large is the people analytics team at NASA? Can you tell us a little bit about how the team's organized?

Sure. So the people analytics team right now, we've got about 10 civil servants and another five or six contractors. So with the way, you know, with a government agency, if you're unfamiliar, you know, you've got a combination generally of civil servants and contractors that do the work. And within NASA, we have right around 18,000 civil servants, but we have another 50 to 60,000 contractors at any given time working on various projects. So within the people analytics team, again, there's about 10 or so civil servants and five or six contractors. But then we have tangential people on the team that may work for other organizations across other human resource organizations across NASA at the various centers that I can pull upon. So they may be doing an HRBP role, an HR business partner role, but part of their work is also to do analysis. I can pull on them to do some analytics at the various centers. So we have, if you add those folks in, maybe another 20, but the core team, it's really the 10 civil servants.

Sure. I might be misunderstanding people analytics because it's not my field. I was wondering if you include things in your research or any of your modeling or anything that are external to NASA that are like STEM graduation rates, things that like down the line might inform NASA's ability to hire people, people that will be available to be hired.

Definitely. I mean, I take a good part in that and looking at external research and seeing what we can utilize. But some of the things we've got a contract with Gartner with their talent neuron. If you're familiar with that, it's basically a platform of looking at hiring and job opportunities across the world, actually, but you can focus on different areas. There are other tools such as burning glass or glass door. The Census Bureau, of course, provides us a lot of information. And then we have our internal NASA external data sources, USA jobs, which is where we post all of our announcements and jobs. We can see what's being asked for across the government or USA staffing, which is the actual location where all the resumes and everything come in. And we get all the information about the candidates and we can look at that.

But looking at those trendings, like you were talking about, a lot of that comes from external data sources, again, like Gartner, burning glass, glass door, some of those places that we can look at. But I'm also trying to look at other iPads, I guess, from which is the education. I don't remember what the actual acronym stands for, but there's a lot of educational type data and they're looking at colleges and universities, the graduation rates, things like that, the demographics breakout. So I'm trying to incorporate a lot of those various data sources to help us inform where we're pulling from. And one of our, and it's going to become bigger because one of our big focuses this year and onto the next several years is the administration's requirement or executive order on DEIA or diversity, equity, inclusion, and accessibility and how we look at it. And a lot of that revolves around recruitment and internships. And are we using the minority serving institutions and things like that? So we are definitely trying to expand our external data sources.

Areas of focus in people analytics

Thanks, David. I'm thinking as well for people not as familiar with people analytics or maybe teams that are thinking about getting started with people analytics, what are all the kind of different areas that you focus on?

Oh, that's a good question. So people analytics can definitely be broken into so many different areas. You can look at, of course, there's some of the basic ones, of course, your attrition, your time to hire, your recruitment capabilities. There's a lot of different modeling or research you can do. Really any particular data scientist that just has common knowledge on those capabilities. So NLP is definitely probably a good tool to have just because a lot of the data we have is unstructured and being able to pull from that. But also understanding various classification models, how you're grouping individuals and understanding various metrics within the human resource area. Those are things. And I mentioned some of those already. Again, time to hire, attrition, the average number of people we have and how often they've left. So it's really just a good way of digging into demographic type information and trying to uncover areas around different concepts within human capital. And again, those fall down to recruitment, talent acquisition, learning and development, attrition. I think those are probably the four biggest areas that you can look at within people analytics or within human capital that people analytics would serve.

Great. Thank you so much. That's really helpful. I see somebody asked on Slido, how much of the people analytics team is focused on data science and predictive modeling versus reports and business intelligence?

It should be a combination of both of those. But within my team right now, everybody does a little bit of everything, unfortunately. We just don't have enough people to do those things. And a lot of that really falls down to our data pipeline and how we're able to access our data. For the most part when I got here, they were primarily focused on reporting and creating some type of visualization, creating some kind of custom report or something like that. Over the last couple of years, I've been trying to break that to where we've got folks that focus on those types of reports in general and then breaking off a group that's looking more at the data analytics, the modeling, creating the various algorithms that we can utilize for extracting skills and competencies or taking a look again at our time to hire models or our attrition models to identify developing some simulations.

So right now I'm trying to get it about 50-50. But for the most part, everybody's doing a little bit of everything. Even sometimes I have to go in there and start looking at some visualizations and create some quick dashboards because it's coming from high up that we need this now. So if I had the money, it would be an equal team of both capabilities. I would probably say about 20 people I would need to fulfill everything we're trying to do and all of those having to count some of those. I would probably break it down to about I think I could get away with five or six visualizers, 10 data scientists, and then the rest of those being some type of data engineer, data architects.

Why should people care about space exploration?

Cool. Thanks for breaking it down like that, too, like a recipe for the team. I see there's so many great questions coming in right now. And one that was asked a bit earlier was, in your opinion, why should an average person care about space exploration?

Well, it's not so much the space exploration. Well, it depends on your curiosity and what's out there and just thinking that that's the biggest thing, just from knowing what's going on in space and knowing your environment. But it's the tangential things that come from space exploration, because as we try to learn how to get to space, how to get to low Earth orbit and high Earth orbit and beyond into outer space, we pick things up. We have to figure things out about how do we handle radiation? How do we handle the lift? What type of software is necessary to handle all of these things? What type of communications are necessary? All of those things provide, as we learn in those things, provide benefit to humankind.

Now, one of NASA's primary missions is to benefit humankind. If you look at our mission, that's one of the statements down there, is that we're not just doing this for ourselves. We're trying to make things better across the world. And it's not just about going to space. It's looking at the different things that we do. So if you're not aware, the International Space Station is actually considered a national lab. We do experiments out there. We conduct thousands of experiments over the last decade. All of those experiments, a lot of them are medical, a lot of them are high radiation, but there's a lot of other things that go out there. All of those have generated some type of patent, potentially, products that benefit back to the economy as well as humankind. So they all, and you see the commercial Velcro, memory foam, all of those things, some of the things that are out there right now were all created because NASA was looking at ways of doing things.

But we have a software transition site to where you can see what kind of software we've open sourced out there, and you can utilize that software for free for, I'm trying to remember, for like three years. And then you've got to make some kind of decision how you want to commercialize that. We've got a lot of different technology transfer groups where a lot of our technology that's being created, we eventually try to get that out into the public. We want the economy to utilize our things that are being created. And there's several companies that have been created based on technology that started at NASA, gone out into the public, and it has provided thousands of jobs and more money back into the economy.

Using RStudio at NASA

Is People Analytics at NASA using R or RStudio products? And if so, what impact has this had?

So, yes, the simple answer is yes. We are using RStudio ecosystem. We've got the RStudio Connect, the Workbench, and Package Manager. We're licensed for that. And it's having a lot of impact as far, at least in my opinion anyway, for how our capabilities of what we're trying to do. When I joined the People Analytics team, you know, again, a couple of years ago, it was primarily a, the team primarily walked around with their laptops. I hate to say this, with their laptops with Tableau on it and said, okay, here's what I did. Here's the visualization and I'll share it with you. So the, you know, I had to work on making these things readily available or easily accessible, not only to our, to the team, but to the end users.

So RStudio was one of those ways of doing that, to be able to create visualizations and reports and presentations and everything else you can do within the R ecosystem and make that easily accessible to not only the, the data scientists and the team, but to the end users to see those visualizations. But at the same time, we were working across NASA to create what we call the Enterprise Data Platform, which was trying to develop that, that pipeline on those workflows to, to make it easy to transmit that data from our authoritative source to a presentation layer. The OCIO team was working on developing a Tableau server instance with an Alteryx no-code, low-code capability. And then connecting that to, to the authoritative data source. That was their start of a Enterprise Data Platform.

Now, it's, that's okay in my mind for those that are more of a data analyst. And those that are more focused on creating visualizations and creating, using data that's already been curated to create those reports and those visualizations. For a data scientist or data engineer who's actually trying to use data that's dirty, you know, and try to clean it up and make it more tidy, we needed other tools to do that. And that's where, you know, utilizing RStudio. And one of the reasons I chose, went with RStudio is because of their continuous growth in how they're using multiple languages, especially Python and R combination of those two. Because half my team was Python and half my team was R. I needed to figure out a way that they could both utilize those things. So we kind of used a combination of RStudio and also VS Code, you know, for some things. But we're able to still create good applications, both in R and Python, and put those on the Connect server, which has provided a lot of value internally within our organization and starting to show some value externally as they're seeing what we're developing and how we're migrating to that.

I wish we were a little further along. We just, again, I don't have enough people, enough staff to get these things to the level I would like. Such much that I'm probably, well, I'm the admin for the R servers right now, only because I'm the one that has the most experience on administrating those. And I need to get the OCIO group to understand, OK, now you guys have to take this over and get people trained up on it. So probably a lot more information than you wanted.

Using Shiny at NASA

It's cool to hear the teams using both RStudio and VS Code, too, and both the Python and R users. I see I'm laughing at your comment or question, Libby, where you said, does NASA use Shiny? I may die of excitement.

We do use Shiny, and it's actually one of the things that it's sought out for by some of our end users, but definitely my team has really come to utilize it quite extensively in developing some of our products. So it's doing very well. I'm trying to get this detail upscale a little bit more, utilizing some of the more modular capabilities. And I just saw that with the Epsilon release, the Rhino system, I need to take a little bit more in-depth look at that to see how we can make the applications, for lack of a better word, more Internet-type ready looking, more robust looking. We started with Shiny as more of a dash, using the Shiny dashboards, and then built one from there to more Shiny applications in developing that. Most of my folks are developers, so we don't have a whole lot of the web developers with JavaScript or HTML, which can really enhance some of those Shiny apps. So we're trying to pick some of that up so that we can make those applications a little bit more robust.

Leadership and people management

Maybe shifting gears a little bit, somebody asked, what's important when it comes to leadership and people management in such a technical company with numbers-driven projects?

Listening and communication. It's probably, from a leadership standpoint, really understanding the needs, not only of your end users, your customers, or who you're delivering your products to, but also understanding the needs and constraints that your team may be having. It's one thing just to go out and do this, but you really have to be there for them to break some of the roadblocks they may be having, but also just provide a lending ear sometimes. I've got a lot of early career people that have come in here, and you need to be their mentor. You need to be sometimes just a coach or a friend or just a listening ear just to get that understanding.

I had a conversation with one of my team members this morning, just as a code review. He was having, and Rachel, I may get back with you on this, we're having an issue with the Dash app on the Connect server that it works well on the desktop, but when we post it on the Connect server, the callbacks run very, very slow. So we were just doing some code review, and he just really appreciated the time with just me sitting down with him looking over things, looking at suggestions. So as a leader, you need to be able to do that, but you also need to be able to have some good constructive criticism on things they need to improve upon or things they need to work on or just learn about, but also letting them know that that's common for anybody in this industry.

Again, the guy that I was talking to this morning, he goes, I'm embarrassed by my code, and I said, no, don't be embarrassed. This is how we learn. This is how we go, and I think it was Hadley that I heard from Hadley, but it was 10 years ago. You go through a million lines of code just to get one good – a million lines of bad code to get to one line of good code or something like that. So it takes a lot of practice to go through your code to get to that level that your code is easier to read or easier to work with and things like that. So never be embarrassed with your code. Just get it out there and work with it.

Again, the guy that I was talking to this morning, he goes, I'm embarrassed by my code, and I said, no, don't be embarrassed. This is how we learn. So never be embarrassed with your code. Just get it out there and work with it.

Administering RStudio servers

Hi, David. I'm a fellow administrator of RStudio servers, and we're looking to transition over to a better architected way if we do RStudio. So I was interested whether or not you run your own in-house RStudio servers or you're running externally, and what do you think – you're talking about transferring this to a mainline support department. What things do you think you're – in terms of feeling comfortable about handing over your responsibilities to them, what do you need back from them? Do you need SLAs or some sort of comfort factor that they will look after your baby?

Yeah, so I'm going to start with the first question. I think it's all on-premise. We do everything on-premise for the most part, unless – well, I'll go back a little bit. It's a combination. It's on-premise in a GovCloud, if that makes any sense. It's a government-secured cloud resource. So I have, in one instance, I've got a contractor in California that handles some of the servers there. I also have a GCP GovCloud instance where I have my development servers there. So we've got them at various locations. And that would be always within that. For the most part, it's going to have to be either on-premise or some kind of secure GovCloud.

Government version of AWS, some kind of cloud. So they're VMs, GCP compute engines or EC2s on AWS. And that's kind of what we're going towards for the most part. We're trying to go more cloud-centric to utilize those type of resources. But definitely there has to be some type of concept of operations and some type of SLA as we move them over to get an understanding of. It takes more to manage a system such as an R ecosystem, and this could be for anything, but in this particular case, than just keeping the operating system up to date. You've got to do more than that, and that's kind of what they're focused on right now. And one of the reasons I took over the administration, because they weren't looking at how do we update the RStudio Connect server or the Workbench server in a timely manner. How do we make sure that we have the right packages necessary within the operating system? You can do a quick instance of an EC2, but there's a lot of dependencies that the R servers require that have to also be installed. They didn't understand that, so I had to walk them through a lot of those things and those dependencies that need to be done.

And then setting up the different versions of not only R that you might use within the RStudio, but the different versions of Python you're going to use, you could potentially use, and how to set up the project environment within the Workbench. All of those things are things that need to be thought about and worked through. And those should be some kind of concept of operations. Who's going to manage those and set up SLA that is going to be done in a timely manner and things like that.

Yeah, I just wanted to sort of ask. I'm sort of in the same position where I'm leading a team, but then I'm also like the administrator of like Workbench and Connect and product. And, you know, so it's a tough balancing act. So I'm curious sort of if you have tips or takeaways for how you balance, you know, being a lot of meetings, leading teams, trying to give direction, big picture, you know, thinking, and then also trying to figure out like, you know, why is DevTools at 11 not enabled in this like our profile for this person that is on my team, you know?

Yeah, that's a tough act to balance. And I feel for you, I understand. Part of it, I finally started to learn how to delegate a little bit and making sure that, you know, especially as I brought on new people onto the team and some of the things that I was working on, I'm able to delegate some of the work being done and just perhaps provide oversight rather than technical work on that. So, unfortunately, I've had to back away from my technical work, such as developing the NLP models or things like that so that I can focus on some of the strategies. So you do have to give up a little things here and there. The other thing is I have blocked out time on my calendar and said, this is time. You know, it's you cannot go over this time. Nobody can put anything on that time so that I can focus on something that I need to focus on, whether it's thinking about strategy or vision, writing up documents or working on the R servers. And I try to do that, you know, sometime, you know, at least an hour a day for some things. And then every other Friday, I block out a big chunk of time. So this is the time I need to work on things that I'm having issues with. But fortunately, I've been able to teach some of the younger, you know, early career folks some of these things and said, OK, now go research this, take a look at this and let's figure out an answer. And then let's get back together, see how we can work. We can work through that. So part of it is just, you know, breaking up your day appropriately and then learning how to delegate, which I've always had a hard time with. I've had a hard time saying no.

Interview processes and predictive analytics

So when I think people analytics, one question that comes to mind is understanding how interview questions and the interview process would potentially translate to later job performance. I'm just curious if that's something that has been looked into at NASA and like whether technical interviews, take home projects or specific questions truly lead to a better performance from a new employee. And if not NASA, if the rest of the group has any input, I'd be curious to know if really anyone's working on that.

So. To a degree, I mean, we started a project to take a look at it was more focused around helping our raters identify executive level individuals. So within the government, you've got your general services group, which, you know, your GS 15 down and on down. Then you have your executive level group, which are those that are appointed. You know, they apply for them, but they still have to be appointed by the administration to that position. In order to get to that to to apply for that, you have to answer a huge level. Well, not a huge. There's. Total of eight questions, I think, and two technical qualification, technical specific and technical general questions. And raters look at those responses and say, OK, this person fully qualified or fully answer that question, or this person just missed the mark on that question. And some of them are more technical. Some of them are also around how you lead and some of your examples there.

So we've had an initiative underway to take a look at past responses to see, can we determine, was this a good response or a not so good response or just a terrible response based on how raters have rated them in the past? It's tricky because now we're taught where it's very subjective in the sense we're trying to evaluate how somebody wrote what they wrote and what they write, actually answer that. Not only to answer the question, but provide good substance to that to that question and did it did it in a way that that made sense and provided value. In other words, like me, I'm not rambling, not a whole lot of rambling, but was very concise and very accurate in what they were trying to do. It's. It was difficult because we came across a lot of issues with the raters, first of all, that the raters didn't all rate the same. They weren't trained how to rate or what to look for. It was just their subjective based on their experience. So it's hard to identify how somebody may have rated somebody really good, but another rater rated them not so good or just average. So there was a lot of discrepancies there. And then you have to think about the ethical and legal ramifications on that. Can we justify the algorithm's response legally to the results that came out? And those are things that we had. So there's a lot of nuances, a lot of things you have to consider. But it's something we are looking at and trying to figure out if there's a way of doing it.

Open source at NASA

And actually, I think I saw a Kaggle content or event just recently about evaluating essays, but primarily they were looking at sixth to 12th grade essays. But the concept is very similar to that, evaluating how they wrote it and the value of not just the quality of the writing, but the content that was in there. Was it a good content?

One was, do you have opinions on open source? Was it scary for people at NASA to introduce it?

Yes, I have opinions on open source. I am a big open source fan. I tend to believe it's – for more often than not, it's probably a lot better. Well, it's easier to understand, easier to work with. And of course, as a software developer myself, it's good to know – being able to see the pack in a lot of times, which you can't see on some of the Kot type products. Was it scary? I think in some cases, for some groups, it was scary. For IT security, it was scary because it's open source and you've got to really take a look at it, don't know what's inside of it. It hasn't been vetted. It hasn't gone through a lot of different security clearances or checks or things like that. So it was scary for them until it became more ubiquitous across the industries being utilized. This was many years ago when open source was just coming out.

The other scary part is maintenance and operations of the open source software. You have many open source companies, of course, have their commercial sites where you can use them for operation. But a lot of organizations use open source because it's less costly to buy the software, but there's still a cost to maintain it and operate it and manage it. And people didn't understand that, and people then were realizing this because in some cases, it was costing them more to use the open source software than a commercial product because they didn't have the staff to manage that open source software. And maintain it. And unfortunately, kind of like I'm doing with RStudio right now, even though it's a commercial product, I started it as an open source. I learned how to manage it and maintain it, but we don't have a whole lot of other people that know how to manage it and maintain it. So we've got to train those folks up, which is an additional cost. But as we get more to the point that we're using it as a more enterprise type capability, that cost will go down because more and more people are utilizing and going there.

Tackling an aging workforce

David, I see someone had asked earlier, how does NASA tackle an aging workforce? A lot of fundamental knowledge may be lost if a new cohort can't be trained before existing staff retire.

Another excellent question, because we do. Man, I was just playing around with some data earlier and looking at the average age of our individuals. And we're up there in the 50-60 area. If you look at the distribution, we've got a lot of – most of our folks are in the 50-60 age group and pretty close to retirement. So one of the things we did in 2009, if I remember correctly, NASA implemented a – they already had knowledge management offices across each of the centers. But they implemented a chief knowledge officer across the agency. And one of the primary objectives of that office there was to help capture information across the agency. So each of the centers do things maybe a little differently. While I was at JSC, anybody that was getting to retire that was – what they would consider a high-level target as far as knowledge, they would interview them and basically write a book on everything that they knew and try to capture that. There's also other management techniques that we – knowledge management techniques we use, lessons learned, trying to capture that across the various organizations, case studies, looking at different issues or things that may have happened and developing case studies on that. We've also started to implement – many different groups do it. Again, sometimes we can be kind of a wild, wild west where each organization is doing some of their own things. But a couple of groups have instituted some very good wikis to where people are capturing this data. Just for that example, they've realized that we have an aging workforce and we need to capture this information. So those things are going to there. So there's a lot of things going on to try to capture from a knowledge management perspective this type of data and making it available and searchable for people that are coming in now and in the future.

Diversity, equity, and predictive hiring

One was, how do you all think about system assessment? Any follow-up comparing a hire's actual performance to the prediction? And then also, how do you think about equity and diversity?

So we're not yet into the complete predictive capabilities within people analytics. One of the things I had to do early on to management is to first get them to understand the difference between descriptive and predictive and try and understand why we needed to upscale or improve our infrastructure, especially with the amount of data. They were somewhat happy with it. Well, we've got all this data here and we're able to use it. Well, kind of you're able to use it. I'm able to tell you right now what's good today based on this data set, but I can't look in the past and I can't use that data to tell you what we might be able to see in the future, and that's where we need to get to more of a predictive model. So I guess the answer to the first question is, no, we can't do that right now, but it's definitely something that is on my roadmap to try to be able to look at those types of things to understand, you know, did we hire, did the people we hire actually pan out, I guess, for lack of a better phrase, and how do you define diversity, equity, inclusion?

This book just came out a couple of months ago, but it's all depends on, well, diversity is fairly easy enough, I think, to understand, you know, is your population that you're working with diverse enough? The key there is diverse compared to what? You know, is it just supposed to be 50-50 all the way across, or what are we looking at? So do we compare it against the population across the United States, across the whole world, across the area, you know, that we're looking at? You know, if I'm at Johnson Space Center in Houston, am I only concerned about the population around that area? Or, you know, so that's a lot of questions that have to be answered around there. And inclusion, more about whether the individual feels like they belong and they're being heard within that organization.

Writing resumes for algorithms vs. humans

You know, I'm just reading an article recently, and it spoke about writing your resumes for an algorithm or AI system, as opposed to a resume that would be evaluated by humans. So I'm really, really interested in finding out, anybody, your thoughts on that whole situation in terms of tweaking your resumes to be evaluated by an AI system as opposed to a human being.

You know, it's interesting because I know that a lot of people have written, well, at least to do that certification process that I talked about earlier where you capture, you're making sure that your resume catches all of the different objectives or requirements within the announcement. That's kind of writing towards an algorithm because you want to capture it because there is a kind of a tool they utilize to make sure that they find those things in there. So a lot of times you've got to make sure it's just basically a keyword-based capability, that that keyword show up in both areas. So that's kind of writing to it. In the end, it's still a human that's got to read through it. And you really need to be able to write to the job and the human is going to look at it and try to get, feel that you really want this position or this is somebody that really would be a good fit. And you can't do that with an algorithm, at least not yet. I don't think you can. But a lot of it comes also back down to the interview process itself. So I don't know. I mean, I'm torn a little bit of both. I mean, I think there's benefits in using the algorithms to help narrow the selection down, but it's still going to come down to a human making the final decision.

Analytics in contract selection

Do you guys use analytics to determine which company gets a contract like SpaceX versus Amazon and stuff like that? Or is there some sort of AI that gives you indicators to go with this sort of technology instead of another type of technology?

Not yet. And primarily because those are protestable issues. So when you do a contract within the U.S. government and a company, many companies apply for that contract. And their selection is made. There are many ways the people who didn't get the contract can protest that. And that is an easy area right now to say I'm going to protest that AI that was utilized. And that would slow things down a lot. So legal has basically said that's not something we should or could be doing right now. But that being said, that doesn't mean we're not looking at it. Not to make the decision, but as an indicator to weigh it, right? That's what I meant. No, there's no AI right now that's doing something like that or ML that's doing that right now. There's probably, depending on the contractors, they've got some spreadsheets out there where they're just tallying up individual, again, raters scoring on that company. And then come up with a way to score. But nothing that's being done programmatically just because of those legal concerns. At least that's what I've been told in the past when I brought it up. But we are, from a human capital perspective, trying to work with the procurement to try to work on some of these things to see how maybe not so much in the contract itself, but on the performance of the current contractor based on their