Computing and recommending company-wide employee training pair decisions at scale... posit conf 2024

Hey everyone, um, yes, I do have a four line mouthful title, but I have a shorter title as well. So today I will be talking to you about a collaboration that I did with my colleague Jordan here and a couple other people at Regeneron.

So I'll be talking about a collaboration that I did, um, at Regeneron in order to compute and recommend community-wide employee training pair decisions at scale via an AI matching and an administrative workflow platform that we developed completely in-house at Regeneron. But as you heard, the alternative title is YouCanDrStrange2 and I will get to why that's the alternative title in a couple of slides.

But first here's an overview of what we're going to be talking about today. So we had a worthwhile challenge, um, and we'll be talking about exactly what that was, but then we addressed it with a scalable solution using a flexible framework for impactful data science.

The challenge: growing mentoring cohorts

So have you ever wondered how impactful companies stay ahead of the game? Well, at Regeneron, upholding and ongoing responsibilities to help improve the lives of others is one of our top priorities. And so we do this by continuously focusing on evolving our people, our process, and our technologies. And so there's a bunch of different ways of investing in employees. But one of the ways that you can do it is by providing growth opportunities. Another is by supporting internal groups that facilitate networking. Another is creating DE&I, so diversity, equity, and inclusion initiatives. And another is enabling the work-life balance, um, that is ever elusive for a lot of us in our careers.

But one of the things that I really want to point out is the career mentoring and coaching or institutional knowledge transfer on a pairwise basis. There are a lot of people who are further advanced in their careers that can help out people who are not as advanced yet, but are hoping to be able to be at that level eventually.

So in terms of the mentoring, we actually had a worthwhile challenge. So our mentoring cohorts have grown substantially every year. So if you think about it in terms of, for example, in the year 2022, if you think about all the mentor and mentee applications that we had, if you multiply those numbers together, it's almost 4,000 choices, possibilities of pairs that could occur. And in 2023, that number grew to 35,000 and this year grew to about 84,000.

So in 2022, the process was done manually, and it's a really grueling, excruciating process to look at all of the different possible pairs and be a matchmaker. So that took half a month for those 3,864 choices. But at that rate of 2022, it would have taken four and a half months last year and almost 11 months this year. And that's probably best case scenario because we're all flesh and blood here. So I mean, it's likely that there would have been a slowdown exponentially increasing the amount of time that it actually would have taken.

But it's too long, right? So trying to do this process across all of the applications and the possible pairings would just be impossible. And the random pairings of just saying, oh, let's stick this person with that person, that would have resulted in a lot of disappointment.

So what we ended up with was a problem. There were too many choices and not enough resources. If you try to do the pairing right, you would be irritated because it would take too long. If you try to just stick people together, ultimately as the matchmaker, you'd be embarrassed because everyone in the company would be angry at you because they're like, why did you do this to me?

So we needed a way to easily consider more people's requests and scalably. So where 20 times the number of possibilities did not equal 20 times the amount of time. So how could we, and here's the Dr. Strange part, and it wrote itself, quickly review the 84,000 possibilities to find everyone's best possible match in far less than a year. How could we, Dr. Strange?

Building MAGNETRON AI

So what did we do? We built a scalable solution. We supercharged Regeneron's resources with AI, making it such that the testable options were far under the amount of resource that we had. And so what we did was we built Magnetron AI at Regeneron, and we've submitted a patent on this. So Magnetron is an acronym that stands for Mentoring and Growing Next Generation Talent at Regeneron Online, Magnetron.

So what this resulted in was the flattening of the amount of time that it took to increase the number of applications and still yield pairs that everyone was happy with. So we dramatically reduced it 10x, and it enabled for cohort sizes to increase without substantially increasing the matching time. It's already been used for four additional cohorts other than the two shown here, and there's one actually happening right now. So it could be used over and over and over again.

So we dramatically reduced it 10x, and it enabled for cohort sizes to increase without substantially increasing the matching time.

A flexible data science framework

So how do we achieve this scalable outcome? Well, we did it using data science, of course. We used a flexible framework strategy in which data science is the key to the impactful AI that we use to accelerate everything. And so if one way of thinking about data science is starting at the end and working backwards, particularly if you want to be impactful, and as we all are data scientists here, if you don't want to waste your time.

So one definition of data science is the use of computational approaches, automation, AI, ML, to enable specific decisions towards desired outcomes. And so what do I mean by an outcome? Well, some sort of improvement in a context or area that you can only achieve by taking an action, by expending resources. But you only expend the resources that you decide to expend and not the ones that you don't. And you only make that decision once you have achieved sufficient confidence by reducing the ambiguity surrounding all of your options. And you only do that once you've gotten enough information about what likely cause-effect relationships could exist. And you only do that once you ask the right questions, attempt to eliminate those gaps that can make it easier to understand potential causality.

So with that in mind, data science and artificial intelligence have this software intersection that define the relationship between the two of them. So whereas data science is using data to figure out which decisions to make, artificial intelligence is simulating autonomous, basically execution of those decisions. So simulating human thinking, reasoning, and behavior. So you can have artificial intelligence systems in which you directly tell the computer what the decision-making rules are, but you can have other artificial intelligence systems that use machine learning in which the computer figures out the decision-making rules by understanding the data itself. And then deep learning, of course, is making computer use multiple layers of thought in order to figure out what the decision-making rules should be.

So in this data science framework, you basically start at the end and you define what that outcome is. And then you work towards identifying the obstacles towards that outcome and then eliminating those obstacles. And that's one of the ways that you can maximize your efficiency and also have staying power in terms of the impact of what you do.

But basically this means that data science is essentially decision science, which really if you think about it is just minimizing ambiguity. So if you just do a little mini quick exercise in your head right now, and you think about if you were to try to pick between the two roads that are shown here, so just pick the right one, like what would you pick? You can't because you don't really know anything about either of the two roads. There's too much ambiguity. And so none of us are confident enough to pick one road over the other. We don't have enough justification to decide on either.

Now if the decision threshold were pretty low, so maybe some of you did pick, and that might be because you're thinking, well, it doesn't really matter, nothing bad is going to happen to me. But if it's a higher risk, if something goes sideways and that it impacts something you really care about, then you're like, whoa, whoa, whoa, I actually need more information. So if you find out, for example, that down one of the roads, there's some sort of danger, well, at least now you've reduced the ambiguity about one of the roads and increased the confidence that you may have about what you should do. So now you have a little bit more confidence. And if there's only two roads, one of them is definitely dangerous, well, maybe you could go down the other road because now you have at least enough knowledge to decide on what you want to do. And so that takes you from being stuck, where you had that analysis paralysis in the absence of sufficient data, to being unstuck. And so that's the goal of data science, to help to reduce those obstacles so you can go from one to the other.

Applying the framework at Regeneron

So I want to walk through a couple examples of using some of this framework to make decisions in other collaborations that I've been involved in. And they're along the pipeline, as we may think of it as a pipeline, in terms of developing therapeutics. So ultimately, at the end, when you have a therapeutic, you want to give it to a patient. But you want to give it to the patient that has a disease that the therapeutic is the most appropriate for. So I'll talk about a couple of these examples and collaborations along this entire pipeline.

So one desired outcome is that genetic disease patients are given the most accurate diagnosis and treatments. So you need to pick between diagnosis A, diagnosis B, or diagnosis N, which could be 2,000, 3,000, 10,000. The decisions are pretty complicated a lot of times. So I won't go into a lot of the details about how I did it, but you can check out the talks that I have on my YouTube channel at youtube.com slash at data-driven decision making. But some of the technologies that I used were ontologies, which are essentially hierarchical vocabularies, embedding-like annotations, which are links between entities and vectorial representations of ontology words as numbers, semantic similarity, which is matching using ontological annotations, and then transitive prioritization, which is guilt by association rank propagation.

So another collaboration, so this is way on the earlier side of the spectrum. We were talking about assigning diagnoses to patients and then saying, well, since this is a diagnosis, then this is probably the appropriate therapeutic treatment. So on the other end of the spectrum, if you're saying, well, what therapy should we even develop in the first place? So one desired outcome of that is disease genes are targeted with the potentially most effective therapeutic modality that we have, and it's at Regeneron because this is where I work. So the technologies that I used to identify whether or not gene A is the right one to target or gene B with a given modality were knowledge graphs, so an interconnected web of data, a Neo4j graph database, which is a knowledge graph server, that's where you can store the graph database, and Cypher, which is the query language that Neo4j uses. And they intentionally picked those names, by the way, because the creators were fans of The Matrix, which I love.

How the decision recommendation engine works

So how did we apply this to scale mentor matching to Dr. Strange, our problem? The real question, actually, was what outcome did Magnetron AI need to be able to prioritize decision recommendations towards? And that outcome really is that employees in need of specific skills are matched with others who are best suited to help them develop those skills. So if we think about it, it's really that the decision is to match each mentee with the mentor that can provide the best help.

So if we start at the end and work backwards, the end is the matching and the best, so matching of the mentee with the best mentor. And so if we work all the way backwards, we need the minimum necessary data to feel confident enough to say this is the pair that we want. And then in order to do that, we need to collect the requisite data. Now previously, they were using Microsoft Form at Regeneron, but we thought that we could streamline and keep the data a lot more consistent if we did this with a Shiny approach, which I'll get into in a bit. But fundamentally, the conversion between the answer and the recommended action is that decision engine that does the processing.

So then the question for you all that you might be wondering is, how does the decision recommendation engine work? Well, fundamentally, it's two steps, filter and then sort. And the filtering is using a lot of the approaches that I mentioned earlier, and the sorting is using the other approaches that I mentioned earlier. And there's two parts to the filtering and one part to the sorting.

So the first part of the filtering is cohort eligibility inference. And you can think of that as like a bouncer at the front door of a club. The second one is compatibility or match compatibility inference. So you can think of that as the bodyguard of a person sitting at a table inside of the club. So one is, are you allowed to even get into the cohort? And the other is, are you allowed to be a match?

So the way cohort eligibility inference works is identifying eligible users, which are users that have any inclusion criteria minus users that have any exclusion criteria. And so the inclusion exclusion criteria membership is defined on the basis of username information, username lists, and reporting structure hierarchy. So these are taken into consideration when a cohort is designed by an administrator.

In terms of the pair compatibility or match compatibility inference, this is asymmetric because you can think of it in terms of in the dating world outside of corporate America, there could be people who are interested in others. But just because one person is interested in 10 people doesn't mean that each of those 10 people is interested in the one person. So there's asymmetry. And so this is what is taken into account with R and Shiny here.

So the asymmetric compatible applicants are applicants with any inclusion criteria minus applicants with any exclusion criteria. And that's from the perspective of each applicant. So then if you take the perspective from each applicant and you have your asymmetric list, if you integrate these two lists, now you have your symmetric pair of eligible compatible people.

And so this is computed in parallel and asynchronously via APIs, as was mentioned a bit earlier. And so it's using the same categories as before for the eligibility for applying to the cohort, but then additional things like years of experience and personal reporting line are allowed to be submitted by the applicants.

And then the ranking part. So what's done here is a space symmetric calculation of skill similarity and activity distance between all pairs that have been applied to the cohort. And then the next step is to do an asymmetric calculation of the rank. So from each person's perspective, how much are the other people good matches for each member of a given pair? And then this is all brought together because then you calculate the average asymmetric rank and then rank that asymmetric rank that has been averaged. So now you have a global rank.

Democratizing the tool with R and Python

So what I ended up doing was democratizing Magnetron AI using R and Python. Python primarily for interacting with the large language model on the back, which I'm guessing some of you are curious about. But overall, the approach to democratizing these analytics are building a knowledge base and then building an API for interfacing with that knowledge base and then building a human friendly interface.

So I use the model view controller paradigm, but you can use others, right? So the model, the view and the controller are essentially the different components where you can have a user open the Shiny app that asks questions of the API, that asks questions of the model, that sends a response back to the API and then back to the user, which makes it easy for a human being to generate very complex queries and analyses on the back end without knowing how to program. And it makes it easier for the FAIR principles to be implemented, which makes things future proof and robust, findable, accessible, interoperable, and reusable.

So I'm going to blaze through the user stories here, but this is really how it works in a nutshell. So someone creates a cohort and then people can apply to the custom URL that the Shiny app provides as a response. And then the Shiny app sends a notification to the approver that they have to approve the applications. And then once the applications are approved, if they are approved, the nominator can go in, click a single button, and then an asynchronous parallel process is triggered that generates all of those 80,000 possible pairs. And that are also ranked via the algorithm that I said before.

And then a reviewer who may know about some of the pairs can provide the additional metadata so that the approvers can take that into consideration when they are either approving or rejecting the nominated pairs. If rejected, the nominators can replace the pairs. And then if they are approved, the notification goes to the mentee, which can also reject. But if it's accepted, then it's sent to the mentor and a final match is provided. And recently, we've also incorporated large language models into this where it generates custom recommendations on the basis of all the information that was provided by each of the members of the pair.

So the result is a user-friendly Shiny app that makes it really easy for people to submit all their information. And it's a single source of controlled structured truth with data access controls in place. And the best part is that it's domain agnostic. So it's usable for the multiverse of possibilities that you could possibly have for matching people for institutional knowledge transfer. And so we've used it for our 400 plus person data science community of practice that I created at Regeneron.

And the best part is that it's domain agnostic. So it's usable for the multiverse of possibilities that you could possibly have for matching people for institutional knowledge transfer.

And yeah, that's really the summary. We had a worthwhile challenge. We hit it with an AI approach. And starting at the end and working backwards is what this team that I was able to work together with over the past two years used to build Magnetron AI. Thanks.

Q&A

All right, so I think we have time for one question. And it has to do with you covered a lot. Where do you think people should learn more, look to learn more about this type of work? About the matching and the tool? I'm assuming the tools that you mentioned on the tool process.

Yeah, so you mean the earlier ones or the, well, so you could look at the YouTube, youtube.com slash at sign data driven decision making to learn more about the earlier stage. There's a paper from my PhD where I talked about transitive prioritization, the guilt by association rank. So that's a potentially good place to start.

All right. So we'll say YouTube is the place to go. All right. So that's the end of this session. Thank you, everyone. Let's give Regis another round of applause. Thanks.