Anubhuti Mishra | Optimal allocation of COVID-19 Vaccines in West Africa | A Shiny success story

story I'm Anubhuti Mishra and today I'm going to be talking about allocating COVID-19 vaccines in West Africa.

So before I get into that, I just want to spend a few minutes talking about my organization. I work with Palladium Group, which is an international development organization. We work with a lot of USAID-funded projects in West Africa and Latin America. I primarily work on this big USAID-funded project called Data Phi, where we primarily work on HIV and COVID-19 programs in Latin America and West Africa.

So we do a lot of data science in public health that involves something like this, building optimization models to building anomaly detection models for data quality assessment, to building risk score models to figure out who in the population is at a higher risk for HIV so that you can optimize your testing resources.

The vaccine allocation problem

So in 2021, we were approached by a couple of West African countries with a very interesting problem that there was an influx of millions of vaccines from COVAX, Gavi, World Bank, USA, into African countries, but suddenly they found themselves with more than enough vaccines to meet their needs, but they found themselves very lacking when it comes to supply chain issues.

Just the infrastructure around sending vaccines across the country, vaccinating millions of adults in a period of less than a year or one and a half years was a huge problem. And we found with a little bit of research and talking to our country partners, we found that a lot of countries ended up wasting vaccines because they were just not able to administer those vaccines before their expiration date.

So this was a very interesting problem, very different from what we were seeing in countries like USA or European countries.

So we thought, okay, this is a great opportunity. Let's build an allocation model. Let's build an optimization model where we'll take into consideration a lot of factors like time to expiration, travel time between distribution centers to vaccine sites, how much staff you have available at vaccine sites to hesitancy and everything. We had a whole model in place.

But then we faced reality.

So this is what the vaccination sites looked like in country. This is a picture of a mobile vaccine site from Cote d'Ivoire or Ivory Coast. And we found that data was being collected across different platforms. Some sites had tablets. Some sites had paper-based data entry.

This is what vaccine storage at site looked like. Some of the better-equipped sites had a data entry person with a tablet who was able to enter patient-level data into an app.

So essentially, we found ourselves in an extremely low-quality data environment. So none of the pillars that our beautiful optimization models could stand on existed. We found that, like I said, data was being collected across different platforms. There was a lot of fragmentation across the data collection chain.

Then we found that the stock data was being collected and managed in an Excel-based tool. There were four different types of vaccines. There was J&J and Sinopharm and Pfizer and AstraZeneca. There was no data on healthcare staff available at site level. So we suddenly found ourselves in a very challenging environment.

Adapting the approach

So this is where we decided that we slightly decided to change the final goal of our work. So we were no longer looking at building an exact optimization model that should give you the exactly correct data on how to allocate vaccines.

We figured that currently the Ministry of Health is just haphazardly using whatever information they have at hand more often than not. It is just based on what is the usage rate at a vaccination site or eligible population. They are not looking at any other factors. So we decided to use some open-source data, pull in whatever proprietary data we could get from the government, and combine all of that to create a vaccine allocation tool.

So this is a list of all sorts of open-source data that we pulled into our tool. We pulled in population data from World Pop. There was data on healthcare sites across sub-Saharan Africa. Then we were able to use a bunch of R packages to calculate travel time, to calculate catchment area for vaccine sites.

That combined with a lot of country data. So even the list of data sets listed here were not available at the start. We had to, and we're still working with the country a lot to get these data sets. They started collecting data on healthcare staff available at the site level after we requested them to share that data with us. We are working with them on how to collect these data sets in a routine and a systematic format.

So then this was our approach to creating this vaccine allocation tool, where we would combine a bunch of open-source data with country-specific data and create this tool, which gives you some sort of an estimate for how to allocate vaccines. We are no longer asking the Ministry of Health to look at this tool as end-all, be-all for allocating vaccines. They should be looking at their ground realities and comparing with the suggestions from this model and then making their decisions on how to allocate the tool.

We are no longer asking the Ministry of Health to look at this tool as end-all, be-all for allocating vaccines. They should be looking at their ground realities and comparing with the suggestions from this model and then making their decisions on how to allocate the tool.

Building the Shiny app

So when it came to where to build this tool, we wanted to, and I think all the presenters before me have mentioned that when you are working in public sector, you're often working with extremely low data environment. Cost is always an issue, so you're always relying on open-source tools. So we wanted to build a low-cost, easy-to-use, interactive, and portable app. And then, of course, Shiny is the only thing that came to all our minds.

So we built a Shiny app, and this is what the landing page looks like. It exists in both English and French right now.

So the app lets you change the input language, so that because most of our country staff speaks French, Cote d'Ivoire is a French-dominant country, nobody on the data science teams speaks French, so we have an English version for ourselves.

So this is our Shiny app, and you will see that on some of the other tabs, we still have text in both English and French. That's for our internal reference. This app has still not been rolled out, but okay, yeah.

So the second tab, called Stock Data, essentially just displays all the stock data that we processed from the ministry's Excel-based tool. You can look at what types of vaccines are available, their batch number, their expiration date, number of doses, number of vials available, and we use this information to build the allocation model.

Then we have the distribution network, which displays the distribution network for Cote d'Ivoire. In Cote d'Ivoire, the vaccines come to a national distribution center. From there, they go to regional distribution center, which from there on, they go to district level distribution center, and then from there, they're sent to vaccine sites. We've divided these into separate maps because obviously it was very difficult to view everything in a single map.

So on the left-hand side, you can see regional to district level distribution maps, and then on the right-hand side, you can see the connection between the national level distribution center to regional distribution center.

Then these are the location of all the vaccination sites across the country. So in our input data, we do have the entire distribution network, so which sites are assigned to which districts. This is a list of all vaccination sites, and users can come in here, and they can select or unselect any vaccination sites because we often see that the ministry will roll out during... They'll have periods of vaccine campaigns where they will roll out 50 to 100 additional mobile vaccination sites, so that can be added here, and then when they're done with them, they can be removed from the model.

So you can see users can come in and check or uncheck, and then they can save the list of vaccine sites.

Once that is done, they can go over to the proposed distribution tab. This allows you... The first user input here we have is number of days because of the complexity of the model, we needed to ascertain how many days does the ministry have to allocate these number of vaccines. So they can decide here that we have these current batches, and based on the expiration date, we want to allocate all of these vaccines in the next 20 days or 30 days. The default we've used is 30 days based on the ministry's feedback, and they can save this information and then run the model.

So once all of this information is saved, we click this button to run the model, which will produce your proposed distribution maps. So this takes a few seconds, but this is essentially a linear optimization model with constraints around expiration date, time to vaccinate eligible population. We'll walk through the list of constraints in a bit.

So this is the distribution map that is generated by the app. You can look at the map, and then if you hover over any of the vaccine sites, it'll show you the name, the regional depot and the district depot that site is connected to and the number of vaccines of each type allocated to that vaccine site. So here we can see that this site has been allocated only Sinopharm, so 722 doses of Sinopharm and none of the other ones. Then we can see here this site has been allocated 960 doses of Pfizer. So on and so forth, you can click on any of the site and look at how many doses have been allocated to that particular site.

Then there's a map below where, of course, the above map was a little difficult to read, so you can filter this map on whichever district you want to look at. And here we've selected Abobo East, and you can zoom in and look at the number of vaccines allocated to each of the vaccination site connected to this district distribution center.

So yeah, this is basically the model. Once this is done, users can go over to results tab where they can view all of these results in a tabular format. They can see how many doses of each type of vaccine has been assigned to each vaccination site. They can download these results, share it with their team, and make decisions based on this.

Then we just have a tab for model documentation. But this is the Shiny app for vaccine allocation tool.

Rollout and optimization constraints

We still haven't launched it. Next month we are going to have a training session with our Ministry of Health and their COVID-19 team in-country. And once this is done, we will be transitioning it off to the team so that they can update and use this tool for making decisions, allocation decisions going forward.

So like I said, this is essentially a linear optimization problem where our main focus was our decision variable was to figure out how many vaccines should be allocated to each site and the optimization objective was to minimize wastage and while maximizing the number of people vaccinated in the eligible population. And we ran into, we started with a smaller list of constraints, but over time we ran into, as you work with the data, you figure out some additional constraints.

So it included that we had to rank prioritized batches based on their time to expiration. So not only did we have four different types of vaccines, for each type of vaccine you could have multiple batches. So we had to make sure that the model starts to allocate the batches that expire first before it allocates batches that have more time to expiration. Then we, yeah, we had to, there were a lot of constraints around expiration date, making sure that it's delivered and administered before its expiration date, that there are enough people to administer those vaccines at the site.

Getting the ministry on board

There was a lot of hesitancy from the ministry initially about using R and R Shiny. As a lot of you might know that normally in public healthcare, especially in Africa, countries tend to use the HIS2 platform. So they were extremely hesitant to, you know, work with a new platform, a separate one for their COVID-19 vaccination needs.

But once they saw, what we did to bring them on board was that before we even got any data from the country, we built a mock app. We traveled to the country, we had a training requirements gathering workshop, we showed them the app, they were able to use it and see how useful it can be for their decision making. And that was essentially all we needed to get them on board. And since then they have been extremely active and instrumental in ensuring that we receive all the data that we need.

We traveled to the country, we had a training requirements gathering workshop, we showed them the app, they were able to use it and see how useful it can be for their decision making. And that was essentially all we needed to get them on board.

We have regular meetings with the Ministry of Health folks and their COVID-19 task force to ensure that they are abreast with all the changes that are happening, that we are making to the model. And we are still figuring out some of the challenges with transitioning this tool over to the country team. Like I said, you know, this is a new tool for them. They don't have our programmers on their team. So we are continuing to work with the ministry to figure out those challenges.

I would like to advertise this a little bit since I got a chance, that I am hiring a senior associate in data science. Ours is a small but growing team, and we work on very interesting problems. We don't have huge data sets, but we do have we do build very interesting models. So do get in touch with me if you have any, if you're interested.