With a little help from my friends: Tools for developing & deploying algorithms in the hospital

Hi, everybody. Happy Tuesday. Welcome to the Posit Enterprise Community Meetup. If you just joined now, feel free to say hi through the chat window and maybe where you're calling in from. I'd also like to say, if you want to connect with each other in the chat, that's all for you, too. So feel free to use the chat to meet other people who are joining us today as well.

Special welcome to those joining us for their first time today. This is a friendly and open meetup environment for teams to share use cases, teach lessons learned, and just meet each other and ask questions. These happen on Tuesdays at 12 Eastern Time, not every single Tuesday, but we do have a community event calendar, which I'll share in the chat in just a second here. Together, we're all dedicated to making this space inclusive and open for everybody, no matter your experience, industry, or background.

During the event, we'd love to hear what questions you have as well, so you're able to ask questions on YouTube Live and also anonymously through Slido, which I will share on the screen in just a second here as well, the link, and I'll put it in the chat. To address one common question up front, yes, the recording will be available, and it will actually be ready right away, like immediately as the presentation ends at the same exact YouTube Live link.

At one of the recent data science hangouts, which happened on Thursdays at noon Eastern Time, I was able to get a little bit of a view into the amazing work the team at Unity Health Toronto is doing to bring predictive analysis to different clinical problems and resource problems in the hospital. And I'm so excited to be able to dive into that deeper today with Chloe Pupron.

Chloe is a data scientist with the Data Science and Advanced Analytics team at Unity Health Toronto. Their team uses high-quality healthcare data in innovative ways to catalyze communities of data users and decision makers in making transformative changes that improve patient outcomes and healthcare system efficiency. I'd love to bring Chloe up here on stage with me, our virtual stage here. Thanks so much for joining us, Chloe.

Thank you for the introduction, Rachel, and thank you all for joining. I'm really excited to be talking about tools and insights for developing and deploying algorithms in the hospital.

So actually, as I was developing the slides, there were a few alternate titles I had considered. Another one that I had considered was this one right here, which is how we got our model out of one person's computer and deployed it to the hospital, which I think is an apt description of the things I'll be talking about for the next hour or so.

So I guess for the next while, I'll spend a bit of time going over DSAA, who we are, what we do, where we started versus where we are now, and really just diving into the main content of the talk, which is where are the things that you need for development and where are the things that you need for deployment? And all of this, of course, is in the context of working in the hospital, working in the health care setting.

About DSAA and Unity Health Toronto

So hi, everyone. I'm Chloe and I'm a data scientist for the DSAA team at Unity Health Toronto. So DSAA or rather the Data Science and Advanced Analytics is a health care data analytics group at Unity Health Toronto, also known as UHT. Throughout the slides, there are a few acronyms. I'll try my best to expand them. I think acronyms might be like a data science thing or maybe a hospital thing.

So basically, DSAA's aim is really to work with Spittel, work with our collaborators and also our partners to make better decisions, increase hospital efficiency and improve patient care and patient outcomes. So in particular, because we're working in the hospital, the projects that we work on will require process and change management. So that's why we work directly with clinicians and with administrative decision makers to develop and deploy our solutions.

Now, what kind of solutions do we work on? A little bit of everything. So we have solutions that will rely on statistics, some that focus more on artificial intelligence, some on machine learning, some on optimization, like operations research, for example. Currently, we have more than 40 active solutions at Unity Health Toronto, and these are, for example, solutions for predicting patient outcomes for enhanced clinical management, planning for hospital bed capacity. We have some medical imaging AI tools and we also have some solutions for assignment and scheduling.

Here is an example of some of the things that we've worked on. So right here, I have a screenshot of our COVID dashboard. So here we're reporting are the counts of people who are COVID positive, people who have been exposed to COVID, as well as people who previously had COVID and are now recovering. And in this case, we're showing the counts for the three different sites that make up our hospital network.

So Unity Health Toronto is based in the Greater Toronto area and it is made up of three different hospitals. So we're showing the numbers for those three hospitals.

Here we have an example of our warfarin dosing tool. This is an application for calculating the warfarin dose based on various input features. So warfarin is a medication, so it's a blood thinner and we're calculating the dosing based off different input features. And then we have our emergency department forecasting tool. So here we're counting the number of patients that are arriving to the ED. So on the right hand side, you can see we have our forecasted numbers and we're also reporting the historical numbers. So you can get a sense of how well the model is doing.

So who are the people that work within the data science and advanced analytics group? So our team is actually made up of four different teams. So we have a data integration and governance team. So they are the ones who will take the data and will connect to the database, transform the data in a way that is amenable for data science. And then we have the advanced analytics team. So they're the ones who are taking the data and building a model on top of it. And next we have the product development team. So they're the ones who are deploying the model.

So they're also the ones that are handling any kind of like downstream integrations. So, for example, if you have a model that is supposed to send a push notification, the product development team will be the ones who are taking care of that. And finally, we have the support team who are really the glue that hold everything together. So they're the ones who take care of project management and coordination at both an internal and external level. So because we're collaborating with lots of different groups within the hospital, this is where having the support team is really helpful.

ChartWatch: an early warning system

Okay, so I briefly talked about some of the different solutions that we worked on. I want to kind of dive in and focus on one solution in particular, which is a successful deployment known as ChartWatch. So in August 2020, we deployed ChartWatch, which is an early warning system for detecting patient deterioration. So it's been running since August 2020. And to this day, it's still running in the hospital. ChartWatch was deployed to the general internal medicine ward within the hospital.

I guess, as an aside, why is it called ChartWatch? Really, it's a branding thing. So before becoming known as CSAA, we were actually known as Chart, which is the Center for Healthcare Analytics Research and Training. So one thing you may notice throughout the talk is like I do mention a thing known as Chart, and it's really because of who we were initially known as.

So we developed ChartWatch. It's an early warning system. And really, it's a model that predicts which patients are at risk of deterioration. So what I mean by deterioration is any of the following. So either transfer to the intensive care unit or ICU, death in the hospital, or transfer to the palliative care unit.

So the way that the ChartWatch model was trained is it was trained on about 20,000 patient visits. And these data consisted of laboratory values, vital measurements and demographics. So these are all things that you would typically find in an electronic health record. And the way the model works is it outputs a risk group. So for every patient, you would generate a risk group. So the patient will be classified as either being high risk versus medium risk versus low risk. And then these risk groups are then delivered to different end users. And you have different methods of delivery based on who the predictions are going out to.

So, for example, we have emails. These are sent to our charge nurses, our palliative care team. And what these emails contain is basically a list of patients with their risk group listed. So if you look at the column, the third column from the right, you'll see the ChartWatch risk groups are included here. Another method of delivery is updates to the front end tool. So here we have a screenshot of our electronic sign up tool, which is typically used by clinicians. And if you'll notice here, the screenshot has a column for ChartWatch. So once again, the third column from the right, you'll notice there is a ChartWatch column and this contains the ChartWatch risk groups. And in this case, the risk groups have been color coded. So high risk group color coded as red, low risk group as green and the medium risk group as yellow. So we're following the traffic light system in this case.

And then finally, the last method of delivery is alerts that are sent to phones. So for patients that are flagged as high risk, we actually sent out an alert to clinician phones to let them know that the patient is at a high risk of deterioration. So going through the different methods of delivery, really, they're varying the level of urgency based off the end users that they're going to and based off the information that's being communicated.

From before to now: the development journey

OK, so I've gone over ChartWatch, but I have this slide just to really pause, take a step back and reflect on how we got from where we were to where we are now. I should probably title this slide back to the past because we are going to the past. So just to give a bit of context, this is where things were at before. So before we had scripts that were running from one person's laptop. It works, but lots of ways in which this can fail. So, you know, what if this person is sick? What if they go on vacation? Most of the laptops we use are Windows laptops. What if we get that Windows update? You know, like the laptop will be out for a while. We have no logging. We have no development environments. That's how things are before.

Now, where things are at now. Well, you know, things are much better. We have service accounts. So basically, this removes the reliance on one person. We log all the things. We have staging versus production environments and so much more. So how do we get from before to now?

I'd say really it's a combination of hard work, trial and error, lots of talking, talking to the right people, figuring out what works and also what doesn't work. And I'd say out of all the tools and findings, I think we can summarize them into two categories. So we have the tools for development and also the tools for deployment. So let's dive right in. What do you need for development?

Tools for development

So the first thing that we need for development is really connections to databases. So within the hospital, there are various data systems. So, for example, we have a different system for labs, a different one for surgical procedures and so on. So in order to help the data science team work better, we developed a package called ChartDB. So ChartDB is an internal R package to interact with hospital databases.

And with regards to connecting to databases, we wrote a bunch of connection functions and they all follow the same pattern. So, for example, if I want to connect to database A, I just need to give us input, my username and my password. If I want to connect to database B, I also give us input, my username and my password. And if I want to connect to the hospital's enterprise data warehouse, or rather the EDW, I would give, as you expect, my username and my password as input to the function.

So internally, this is actually what's happening. Internally, we have a bunch of retries because, you know, the connection may not work the very first time. We also have a bunch of checks, so making sure that the connection is successful and that what's being returned is what you expect. So ChartDB offers connection functions and it also offers other utility functions. So, for example, I have this get patient ADT function right here. And what ADT stands for is Admit Discharge Transfer. So, basically, when I call this function, it will return a list of patient movement events.

All right, so we have database connections. Another thing that we need for development is an environment that lets multiple people collaborate on projects. A really reproducible development environment.

So in order to help with that, we ended up using renv. Here I have a link to renv and here's a screenshot of the documentation. As I said before, at the end, all of the links will be included in the slides. So renv is an R package for R dependency management.

And how are we using our project? Well, here's an example of our project structure. So, you know, we have our main folder, we have a bunch of files, a bunch of subfolders. And really, I want to bring attention to that last file at the end, which is the renv.lock file, which is where all of the renv magic happens. So this is what the renv.lock file looks like. And basically what it does is it specifies the different packages that your project relies on, as well as the different package versions. So, for example, here, my project relies on deep IR and I'm using version 1.0.7.

So, you know, I could then give this renv.lock file to a teammate and then they can then ensure they're downloading the same packages as all the same package versions to contribute on the project.

Okay, so I've highlighted a few things for development. So I have my database connections, I have my reproducible environment, which is facilitated through renv. Another thing that I need is functions and utilities that are reused. So really what I mean by that is package-based development.

So here I have a link to the R packages textbook, as well as a screenshot of documentation. I highly recommend it. So really, why write a package? There are a bunch of reasons, but I'd say that for us specifically, what we found the most relevant was being able to share code and knowledge with others, as well as having no more copy-pasting. So I find that as I'm developing, if I'm copy-pasting the same bit of code over and over, kind of like, you know, it rings an alarm. You're like, maybe I should take a step back and think of what I'm doing.

So if we revisit the project structure that I introduced a few slides ago, this R subfolder right here is where you have all of your package functions. And then there is the description file right here, which is where you would define the attributes of your package. So for example, this is a description file that I have for ChartWatch. So, you know, it's listing out a bunch of attributes like the package name right here. So it's ChartWatch. The package version 1.11.10. And at the very end, I have the imports right here, which specifies the packages that my package relies on. So the dependencies.

Okay, so our development checklist is, we're adding more to it. So we have our database connections. We have an environment that allows multiple people to collaborate. We have package-based development. Another thing I'm going to add is environments, environments, environments. So the reason I say the same word three times is because we actually work with three different environments. We have the development environment, the staging environment, and the production environment.

So the development environment, this could be your local computer. But in our cases, we usually use a development server. So what our development server includes is GPUs, which allow for any projects that require deep learning. And also, the development server works nicely with ChartDB. So that means that anyone who wants to develop something, you know, they're guaranteed that database connections will work for them. So they'll be able to access the data that they need for their projects.

Next, you have the staging environment, which is as close to the real deployment environment as possible. So for example, when we need to make updates to ChartWatch, we would first apply it to the staging environment to make sure everything is working as we expect it. And then finally, there's a production environment, which is where things actually get deployed.

Tools for deployment

So actually, before I dive in, I'm just going to take a small intermission so I can reflect a bit on ChartWatch's deployment. So ChartWatch was deployed on August 2020. But it was actually first silently deployed in November 2019. And I guess the reason for this large gap is because initially ChartWatch was supposed to go live in early 2020. But then the pandemic happened, it affected plans, the hospital's priorities shifted at that point, you know, they wanted to focus on containing the rising number of COVID cases in the hospital. And then, you know, ChartWatch kind of like took a bit of a backseat.

So just to recap, there are two folders. I start with 00. Interesting. At the very bottom, there is that error check.py file. So there is Python code here. And then also I have my .rproj file at the very bottom. So there is R code. And actually, if you look at each of these different subfolders, you'll find a mixture of R, a mixture of Python. I believe we also had some Java code at that point.

Okay, let's try. I think it should be over. I can't hear it.

Yeah, so, actually, just want to address, like, there are some really great, great points in the chat. I really like what you guys are saying about branding. So that's kind of, like, one of the things that we try to do with our, like, our ChartDB package. Yeah, definitely like that. Yeah, going back to how things were before. So when we first tried to deploy ChartWatch, we had a mixture of Python and R scripts and also some Java. And the way we were running everything was we had this complicated thing.

Yeah, so, yeah, the way things are running is we had a mixture of Python and R scripts, and we had this cron job, which was calling different bash scripts, which was then calling different Python and R scripts. And then I guess where we were, the biggest red flag is that our test environment was our staging environment, which was our production environment. So, for example, in ChartWatch, one of our outputs was an email which contained a list of patients with their risk group. And you can imagine, like, as I'm developing this, I would send that email to my email and, you know, make sure things look as they should be. But because our test environment was our staging environment, was our production environment, when it came time to finally deploy, I would have to, like, comment out the line that had my email and then set it to be, you know, whatever test email we had. So, you know, like, if I'm very diligent, if I'm very careful, things will work out. But this method is very prone to errors.

So, with that being said, what are the things that you need for deployment? So, the first thing that we really need for deployment is authentication. So, all of our deployed applications run on RStudio Connect, which is actually now Posit Connect. So, one of the nice things about Posit Connect is that it interacts with the hospital's active directory. So, what this means is that users can authenticate using their hospital username and password. So, for the developer, you know, we don't need to keep track of an extra server username and password. And then for the end users, in order to access applications, they could just log in with their hospital credentials. And a nice thing about being able to interact with active directory is that we can just use the existing groups to manage permissions.

So, for example, if I have an application that I only want to give access to the SAA, so like the data science group, I could just use the existing active directory group to enable that. Another thing that RStudio Connect, or rather Posit Connect, allows us to do is to schedule scripts. So, we run our applications from service accounts, which is basically a non-user account in which we can run automated services. And it's this combination of automatic scheduling and using a service account that really makes it so that deployments don't rely on one person.

And it's this combination of automatic scheduling and using a service account that really makes it so that deployments don't rely on one person.

So, you know, this is how things were before we had this cron job, which would call different bash scripts, which would then call different Python or R scripts. But now that we're using Posit Connect, we can leverage a lot of different things.

Sorry, the screenshot is a bit small, so I listed all the different things, all the different tabs on the side. So, you know, we have the application information, we have the access tab, which is what we use to specify, for example, who can access a specific dashboard. We have the runtime settings, we have the scheduling settings. So, scheduling settings is what we use to schedule our scripts. And then there's the tags, really for organizational purposes. So, for example, all the ChartWatch scripts would get tagged as being ChartWatch environment variables and logs.

So, we're adding a few things for deployment. We have authentication, we have scheduling. Another thing that's useful for deployment is being able to know when there's a downtime. So, for example, here, this is a screenshot of one of our Slack channels where, you know, we received an alert, you know, ChartWatch is down, for example. So, in order to enable this, we created Jarvis, which is an R package for helping us monitor our production applications.

So, Jarvis is the IRONMAN's robot assistant. Yeah, that's where the package name came from. Yeah, so we use Jarvis for a few notifications. So, we can use it to send email alerts. So, in this example here, you know, Jarvis sent email letting me know that there's an issue with the COVID dashboard. As shown previously, we can use Jarvis to send Slack messages, like here. And we also use Jarvis to check the health of systems that we depend on. So, for example, if you want to check a specific database, right, I can check that my connection to the enterprise data warehouse still works. And I can also check that I can access different file systems with Jarvis.

So, on top of using Jarvis for different notifications, we also developed a few downtime protocols, which are pretty project specific. So, for example, with ChartWatch, if ever there's something that goes down, we would send this kind of email to the hospital, to all the hospital users. So, we have a specific email template that we use here. And this email template actually follows the one that IT uses. So, you know, it's color coded at the top where, you know, red is for unplanned downtime, blue is for planned downtime, and so on. There's a list of things that we need to specify. And this will go out to all the hospital users. So, like, not all projects will require something like this. But something like ChartWatch, which is embedded into clinical care, does require something a bit more robust. Basically, like, when things go down with ChartWatch, it's not just the DSA team that needs to be notified. It's other people within the hospital.

All right, so you have authentication, scheduling, downtime. I'd say another thing that you need for the deployment is a secure way to download internally developed packages. So, to give a bit of context, around the holiday time last year, we started seeing these kinds of messages in our Slack channels. So, you know, like, not able to receive emails, you know, check log4j. Sorry, running late because I was on a meeting with IT because of log4j. If you use PyCharm, be careful. There's a log4j file in there. I found log4j files in my VS Code folder. There's something going on with log4j.

The log4j, for those who don't know, it's a Java-based logging utility. And in December 2021, there was the log4j flaw. And basically what happened is this could allow malicious users to access internal networks. And I guess just to highlight the severity of the log4j flaw, like, here are some headlines that were coming out around that time. So log4j is a pervasive vulnerability. Update your devices now. A hole in a popular piece of code is an open window for criminals. The log4j software flaw is endemic, according to the Cyber Safety Panel.

So really, this whole thing about log4j is to motivate the next point is that we need to limit who can access the hospital network. And this is where something like Posit Package Manager comes in. So Posit Package Manager, also formally RStudio Package Manager, is a repository management server. And what it allows us to do is download packages while being disconnected from the Internet.

So if I go back to renv, which I mentioned a few slides ago, remember we had our renv.lock file, which specifies the different packages and versions. So here it specifies how the package is being downloaded. So instead of being downloaded from, say, GitHub or from CRAN, we're downloading from RSTM, which stands for RStudio Package Manager, which is also Posit Package Manager.

Implementation plan and clinical workflow

So a few more things that we've added for deployment. So we just added a secure way to download internally developed packages. One more thing I want to add is an implementation plan. So, so far, all the things that we've discussed previously, I guess you can apply to data science in general. For this implementation plan, it's probably a bit more focused on the health care setting and on deploying within the hospital.

So just to recap how ChartWatch is delivering predictions, they're going to different end users. So we have our emails that are going to charge nurses, the palliative care team. We're updating this electronic sign out front end tool that clinicians access. We're sending alerts whenever there's a patient that's flagged as high risk. This is going directly to clinician phones. And really, if you like, just looking at the way things are delivered, there's a lot of people involved. So you have like your church nurses, your palliative care team, you have I.T., you have residents and physicians.

And it's with that in mind that when we when we came up with what we put together, the implementation team, we wanted to make sure that people from all of these different end user groups are being represented and included in these implementation team meetings. Right. So we made sure we had people from general internal medicine, from the intensive care unit, from palliative care, from clinical informatics, from I.T. and then like from the data science team. Just want to highlight that collaboration is really important here. And also the whole idea for ChartWatch actually came from a general internal medicine doctor. So they were the ones that are leading the project throughout the whole course and really making sure we were going from development all the way to deployment.

So one of the things that the implementation team was tasked with doing was coming up with an intervention. So after ChartWatch flags a patient as being high risk, what happens then? So the implementation team develop a clinical workflow. This is a clinical workflow looks like. I won't go into too much detail, but I will point out a few things. So, you know, when a patient is flagged as being high risk by ChartWatch, there are a few things that need to happen after that. So one of them is placing orders to increase vital sign measurements, reviewing recent labs, reviewing recent medications and recent imaging orders. And really all the things that you find in this clinical workflow are all things that a clinician would normally do if they suspected a patient of deteriorating.

So when developing this, this checklist, this clinical workflow, it was really important to make sure that it wasn't disrupting any existing processes. So you want to consider existing resources. You don't want to reimplement a workflow that already exists, for example. And also the alerting notifications all fit within existing processes. So, for example, when it came to the timing of emails, some of the emails that we sent out is sent to the charge nurse and this email is sent out twice a day. And they're sent at a time that makes the most sense with how the shifts work. So once again, we don't want to disrupt how things are working currently.

And with regards to the clinical pathway, really ensuring that there are time targets. So, for example, after ChartWatch flags a patient as being high risk, one of the things that the clinicians must do is increase number of vital measurements within 24 hours of the alert being sent. So having this 24 hour time target makes it that we can then go into the electronic health record and then measure whether or not this is happening. And, of course, leaving room for clinical judgment.

And so when working on the implementation plan, we include a signed deployment period. So basically, this is where your entire pipeline is running, but the final outputs are silenced. This is useful for identifying bugs and addressing unexpected changes, such as change to how the Troponin lab test is done. So the Troponin test is one that's commonly used to diagnose heart attacks and other heart related problems. So during the deployment of ChartWatch, like the way this is measured got changed. And this is something that we have to address. So we have to address that in our data extraction code and also in our data processing code. Other changes that can happen can be due to deploying at the beginning of the pandemic. So, you know, during that time, there's lots of rooms that are being made available or unavailable based off, you know, the patients are testing positive for COVID. Patients are being moved to completely different units in the hospital just based on capacity. So this has to be addressed in our data extraction code.

Another thing that the signed deployment period was useful for was for catching bugs. This is a really interesting bug that we encountered. I'm guessing that most people that are watching this talk are R users. So, you know, I can ask the following question, how are missing values represented in R? So I think most of you know it's an R missing values are represented as NA, which means unavailable. Now, next question is, what's the chemical element for sodium? I think some of you can see where this is going. So the chemical element for sodium is also NA.

So we work in the hospital. In the hospital, there are blood tests that are done. And, you know, one of the things that gets measured in blood tests is sodium. So when we silently deployed ChartWatch, we noticed that there were no sodium tests that were being measured. This is an example. This is a graph that shows kind of all the electrolyte counts. So in blue is sodium. And if you notice here, the blue is set to zero. OK, like this doesn't seem right. Like what's going on? So when you went and checked the code, what we found was that the R code that we were using to read some of the lab's data was interpreting all of the sodium tests as being NA not available. So yeah, this is what happened. The fix was actually pretty straightforward. It was just like updating one of the function parameters to not read NA as not available. It was just a very interesting bug that we ran into.

So the chemical element for sodium is also NA. So when we silently deployed ChartWatch, we noticed that there were no sodium tests that were being measured. So when you went and checked the code, what we found was that the R code that we were using to read some of the lab's data was interpreting all of the sodium tests as being NA not available.

So once we were done with the sign period, it's time to go live. So actually for us, we had a pilot phase in which we first deployed to two general internal medicine teams only. And during that time, we had weekly meetings which were used to debug any issues we ran into, go over the patients that were flagged by ChartWatch, you know, check like, does it make sense or is there anything weird or unexpected? So that was the approach that we took for going live. And then we really did in a phased approach to start off with two teams and deploy to the rest of the teams.

And really throughout this whole process, so throughout the pilot phase, throughout deployment, throughout, you know, developing the clinical workflow and user engagement is important. This is not something that could have been possible with just a data science team. So when it came to developing the clinical pathway, we really relied on the general internal medicine team, on the palliative care team, on the nurses we were working with to develop that pathway. And also when it came to developing education and training processes, that was, once again, a highly collaborative effort. So yeah, so things need for deployment, implementation plan, super important.

Monitoring in production

One last thing that I will mention for deployment is something about monitoring. There's a question mark here because, you know, this could be a whole talk and we're already nearing the end of this talk. So it is a whole talk. There's plenty of other people that have spoken about monitoring. So I would redirect you to watch these amazing talks. Lots of great insights. With the time remaining, I will mention a few things specific to chart watch, though.

So a chart watch, one of the things that we want to look at is model performance. But that's a bit hard to do once the model has been deployed.

OK, so we just backtrack a bit. So with chart watch, one of the things that we want to look at is model performance. But that is difficult to do once the model has been deployed. So clinicians, one of the things that they asked for was a target positive predictive value of 0.33. So what this means is that for every three alarms, at least one of them should be correct. But that is kind of hard to measure once you are deployed and in production.

So what I mean by that is, let's say chart watch flags a patient as being high risk and sends out an alarm. And then that patient doesn't deteriorate like they don't transfer to the ICU. They don't die. Does that mean that the chart watch prediction was incorrect? Or does that mean that the chart watch prediction was correct? And then because clinicians received that alert, they were able to intervene and prevent that outcome from happening. So, yeah, in deployment, we we get these cases where there are these false alarms, which we're not sure if they're actually false alarms or if they're averted outcomes just based off this successful intervention.

Yes. Another thing we measure in chart watch is the daily number of alerts. So on average, we're sending two alerts per day, which is a reasonable number that the clinicians are happy with. There are some days where there are no alerts and there are some days where there are lots of alerts. So, for example, I think there's like nine or eight alerts sometime in like April or May.

So, yeah, one of the things that we were really concerned with was make sure there weren't too many alerts. So we worked with the clinicians to ensure that we were minimizing alerts. So right now, the way things work is that, you know, a patient will receive an alert if chart watch classifies them as being high risk. We added some additional rules on top of that. And those rules we develop with the help of the end users. So, you know, we don't re-alert a patient unless it's been 48 hours since their previous alert. You know, this just ensure that we're not alerting for the same patient every hour, for example. And we added this additional rule, which is that we don't alert a patient if they just came from the intensive care unit. Because if a patient is coming from the ICU, you know, like the clinicians already know that they may not be doing too well. And it's expected that, you know, their vital measurements, their labs might be looking a little off because they just came from intensive care.

The last thing I will say about monitoring is regarding clinical adherence. So how do we know that when a patient receives a chart watch alert, the clinician is actually doing something and they're not just ignoring that alert? So one of the things that we look at is the number of vitals that are measured after an alert is sent. So we do that because as part of the clinical workflow, one of the things that the clinicians have to do when they receive a chart watch alert is increase the number of vital measurements. This is something that we can easily measure just by looking at the electronic health care record, looking at the number of vitals that are sent after an alert has been sent. I mean, it's not a perfect measure because the number of vitals can increase for reasons unrelated to chart watch, but it's a proxy and it gives us a good enough idea of clinical adherence.

Summary and Q&A

So this brings us towards the end of our talk. So let me just summarize some of the things that we talked about. So we looked at the different tools for development and deployment of an early warning system in the hospital. And so on the development side, the things that we found useful were database connections, reproducible environment facilitated through renv, package based development, the different development staging and production environments. And then on the deployment side, we had authentication and scheduling, which were facilitated by Posit Connect. We had downtime protocols, a secure way to download internally developed packages with the help of Posit Package Manager. We had an implementation plan in which we're working with our different collaborators, our end users. And then we had monitoring at the very end. And, you know, all of this would not have been possible if it weren't for collaboration. I think it's something I keep saying throughout the talks that collaboration is really important. Developing and deploying chart watch in the hospital is not something that we could have done on our own. I was really by working with a great team and a lot of different people within the hospital that this was possible.

Developing and deploying chart watch in the hospital is not something that we could have done on our own. I was really by working with a great team and a lot of different people within the hospital that this was possible.

So this brings us to the end of what I prepared. And as I said before, the slides will be available. And at the end, I do have the list of all the papers, talks, blogs that I mentioned. You can all find them at the end. Awesome. Thank you so much, Chloe.

One from the beginning, which we touched on for a second, is can you talk more about the development process? What came first? Were there steps dependent or parallel or did these changes happen simultaneously?

A lot of things were happening at the same time. So in the talk, I think I mostly focused on the implementation team and all the things that the implementation team was doing to get this from development to deployment. But there were different sub teams that were working on this. So there was the evaluation team, which is, you know, they were trying to address the question, like, how do we ensure that the ChartWatch deployment is successful? So they're the ones that are coming up with like, you know, we should measure vitals because you've got to measure clinical adherence in one way or another. And there was the modeling team, which is, you know, mostly the data science people and IT that are focused on actually building the model. So different teams working all at once. I see the way this all started is really from the general internal medicine doctor who came up with the idea first and came to the DSA team and said, like, hey, like, it would be cool if we had an early warning system. And then it's kind of like all snowballed into this.

Thank you. I see somebody else asked, and I guess this is a follow up here, but did the team come first or did you architect these workflows and then build the team?

So like the DSA team, or rather at that time we were the chart team, like we did have the chart team first, so we did have a data science team. I feel like the different groups, like the implementation team, is something that came as we progressed, as we started working towards developing the tool. So we had all of the building blocks, we were assembling it together, that kind of happened a bit later.

So on the topic of the team building, then, can you speak to any lessons learned from building four teams within DSA? What did it take for leadership or management to take on a big scale project like ChartWatch?

Yeah, that's a great question. So when we first started off, we did not have the four team structure. It was, we just had data scientists to do a bit of everything. We found that, you know, there were certain people who were a lot more knowledgeable in one aspect. So I think it just made sense to split it up into this four different teams. And then within each of the teams, there is a director for each of these teams. I think it makes the management and coordination a lot more streamlined. So as a data scientist, I can focus on doing data science work. I don't need to think about all the other unrelated things.

Absolutely. And I see Rebecca asked a question over on YouTube. What is going on under the hood for those alerts that you showed us? Rebecca just mentioned, how do you begin with either functions for emailing or database status?

Yeah, yeah. So I guess more specifically, when we're checking the database status, we are just checking that connection can be made. So, you know, I just try to connect using my usual connection function. If that doesn't work, then I know that something is going on. With regards to actually sending out the alerts, for sending Slack alerts, we are working with a Slack API to send that. And then for sending out emails, there's something you can do. There's a bunch of R packages for that, actually. So the one we use is sendmailr. This is what we use to send out our emails.

I see Ezra also asked over on YouTube, how early in a project do you typically shift to developing an R package?

Oh, that's a good question. That's something that we sometimes struggle with, because I guess you don't want to do it too early, because it might seem like you're kind of like over-engineering things. So I think for us specifically, for ChartWatch, it was... So we first deployed in November 2019, and then everything was put on pause. And it was over the summer that we decided, okay, let's just build this all as an R package. So I'd say by that time, we had all of the requirements all mailed, and we all knew this is what the alerting schema would look like. These were the different groups that we would be interacting with. So I'd say wait until you know what the final requirements would look like before shifting to developing an R package.

I see Toyin just asked, how long did implementation planning take? Large stakeholder groups can be unwieldy.

So the idea for ChartWatch, I believe it initially... The idea initially started back in 2018. And yeah, large stakeholder groups can be unwieldy. It's lots of schedule managing and trying to find a time that works for everyone. But I mean, I think the sort of things that we want to make clear is if you want this to actually be the point implemented, you need to be committed to working towards it. You need to be able to show up to our monthly meetings or our bi-weekly meetings. So yeah, it was hard, but we managed to get it working. And I think a lot of credit goes to the support team who was able to make all of these meetings happen and ensure that the project coordination actually followed through with that.

Jumping back over to... Actually, somebody asked this, I think, over on LinkedIn. How does DSAA fit into the overall organizational structure?

So we are directly part of the hospital. So I guess when we started off, we were a bit more research-based. But then I think it was in 2019 or 2020, we officially became part of the hospital. So advanced analytics is a priority for the hospital.

I know a little bit ago you were mentioning the idea for ChartWatch came from one of the clinicians. Mark asked, in what way are clinicians involved in the development process? How do you prioritize what you work on as a team?

Yeah, so clinicians are completely involved. They're involved from day one. Our project intake model is very clinician-driven. So it's not the DSAA group that comes up with the ideas. It's the end user that comes to us with an idea, and then we work together. So in this case, we have Dr. Amol Verma who came to us. So like, hey, I want to develop an early warning system. Let's work together into doing that. So yeah, clinicians are involved from day one.

Another question was, how did you succeed in gaining the confidence of decision makers in the application?

I think for this is, also, we leverage the clinicians we were working with. So Amol was really useful in talking to the clinicians and telling them like, hey, the ChartWatch model is something that is reliable, that works. I think that also the education piece was useful in, education and training. So it was useful in getting the different end users accustomed and used to ChartWatch. And also, when we first deployed, we did have regular meetings. So it was a good way to kind of get like a vibe check, make sure like, hey, how are things going? Are people still like receptive to that? So just checking in regularly and ensuring that, you know, everyone was happy with how things are going.

I see another question and I might need help with an acronym here. How do you deal with the medical device regulation? If applications, for example, ChartWatch fall under the category of SAMD?

Yeah. So I believe ChartWatch doesn't actually count as a medical device. So the way that we deployed, this was all under the umbrella of quality improvement. So I know that when we, we had to write a few like, we had to write a bunch of documentation. I always found their quality improvement.

A question that I had is if somebody maybe works in a hospital today and is inspired by the talk that you gave and is thinking like, I want to get started and I want to do this at our hospitals.