Novo Nordisk's Journey to an R based FDA Submission

the Director of Life Sciences and Healthcare here at Posit. Very excited to bring to you today the ninth installment of our Life Sciences Series. Previously, we have highlighted Roche, BMS, Merck, GSK, Novartis, Johnson & Johnson, and others. And you can find these Posit webinars linked below. And today we have an exciting webinar featuring the work of Novo Nordisk.

We're going to highlight the journey to an R based FDA submission. And we'll kick off the presentation today with Ari, the Principal Statistical Programmer. We'll then pass it over to Stephan, Statistician. And then we'll end with Anders, Statistical Specialist.

It's such an exciting time for open source drug development. From 2017 to 2020, we had important things that had happened. R and Pharma, the R validation hub, the Pharmaverse. And then into 2021, we had an R pilot submission to the FDA. And now a little over a year later, we have a full submission in R by Novo.

And it's so exciting to see the pharmas are migrating to an open source backbone for their clinical reporting. And what sets Novo apart is the depth for which they're tackling this effort. The change management from a multilingual statistical computing environment to the package management and validation and to the end user support and education. For example, they even have their own internal R podcast. And all the while contributing to community efforts like R and Pharma, the Pharmaverse, the pilot submissions. And so it's with great joy to bring you this session today. And now I will pass it over to Ari.

Thank you, Phil. We're so excited to be here and to be able to tell our story of all the ups and downs we've experienced in the past few years.

Before we begin, I just have to say that this presentation represents the view of the office and not necessarily New Orleans. And any of the regulatory communication that we'll show here will not be applicable to all projects. So please communicate with the appropriate health authorities about your submission.

Without further ado then, earlier this year, we submitted our first R-based submission to FDA, PMDA and other health authorities around the world. And this happened early in 2023. And we have multiple learnings and feedback from authorities that we will share later on in the presentation as well.

Incidentally, this year Novo Nordisk is actually celebrating its 100th anniversary. And Novo Nordisk started in 1923 as Nordic Insulin Laboratory. And it was at a later merger with Novo Industry that the current name Novo Nordisk was formed. Here in Novo Nordisk, we focused on treatments for people living with serious chronic diseases. And we have a strong history in diabetes and now in obesity for which you may know the company for.

Biostatistics and the submission scope

In biostatistics where we three sit, we work with the first three steps shown here. So we work with the clinical trials, data and evidence generation and approval activities. In the evidence that we submit to authorities, we include STDM and Adam datasets that follow industry standards such as the CDISC. We submit programs, documentation, analysis, tables, figures, listings, you name it. And what we're going to share today is our story of getting to do all of this with R.

Now going back to the beginning, then there's one thing we really, really want to highlight and one thing that permeates everything in this presentation. And we put this quote on the slide, that the goal is the transformation process that is revolutionary in result, but evolutionary in execution.

that the goal is the transformation process that is revolutionary in result, but evolutionary in execution.

Or in other words, it's more a matter of taking steps rather than thinking about how big those steps should be. Now, back in 2015, we all joined the biostatistics in Novo to an environment where everything was dominated by the SAS programming language. We loved R and we quickly noticed a huge R shaped hole at Novo. And to our surprise, Gibbs and version control systems were also absent and we were not the first to notice this here.

So we got R installed on our local computers and we used it every now and then for these one-off non-critical tasks.

In 2017, 2016, we started to introduce R to management and our stakeholders and colleagues for non-GXP related work, where we were showcasing Shiny as an incredible powerful framework for data visualization that you just couldn't get anywhere else. We had a Shiny app for data exploration, the bottom one you see here, and one for assistance in pharmacokinetic and pharmacodynamic data reviews.

We were also using R for crawling SAS logs, comparing stuff and checking stuff. And development of this continued through 2017. And in 2018, we continued to push for adopting some use of R. And the collection of R programs we were using had grown so much that we now combined it into a single package named NNR. And this package was the first thing we put in version control and we also set up pipelines for testing it.

Now, come 2019, where we finally saw a commitment from above to report a trial in another language than SAS. I remember receiving this strategy email and Stephan was sitting at the desk right next to mine. And I showed him this and we literally ran into our manager and said, if anybody's going to do this, it has to be us. And it wouldn't take long until we got the nod to go ahead for this. And we both thought, shit, it's real.

How do we even do this? We don't have a TXP environment for R. We don't have a system for packages. I mean, we don't even have a strategy for packages. So we get down to this and I'll hand it over to Stephan to tell you more.

Building the statistical computing environment

Yes. So at this point, we are still using R from all local laptops without any system basically at all. So we were in dire need of a statistical computing environment.

And actually our initial thoughts regarding this new environment were in terms of stakeholders and the change management of those. So we wanted to produce table figures and listings that were identical to those that were already produced in SAS. Because this mean that there would be no change for the stakeholders regarding the language that we would choose. It would also mean that we could use the existing workflows that we had set up for producing reports that we're going to send to authorities. And we could on the same trial work on both R and SAS.

We also wanted a system that were hassle-free for the end user. So we were trying to cater to our colleagues that were used to programming SAS. So we really wanted a server that they didn't need to think so much about packages, environment as such, but more about getting to the actual programming. So it should also be so that if you've got a code snippet from your colleague, you should be pretty sure that that would also work on your computer, given that you were using the same version of R. And then finally, with this new SCE, we really wanted to highlight version control. So remember at this time, we weren't using a version control basically at all.

We had a few things that we had it on, but not for anything regarding trials. So here we wanted to produce an easy way for initiating a version control for both trial and collaboration work, such as packages.

We quickly saw that these four things would be the pillars of our infrastructure. We really wanted an IDE based on RStudio workflow or the opposite, Workbench, because it was such a great IDE and familiar to so many R users. We did, however, want to make it the server-based solution so that you would simply log in to the server and be sure that you were in common ground with everybody else. Then we wanted to use Git for our version control because that was so familiar with most of the R programmers as we had used it in the past.

And it will also come with great features, such as feature branching for our development stuff. We wanted to use Azure DevOps, both for project management, so for organizing tasks, but also for all of the code repositories that we could share those. And of course, for CICD pipelines. And finally, but definitely not least, we wanted to use the Posit package manager for all things regarding R packages. We wanted to use it to share our internal packages. And we also wanted to use it for creating subset of packages that we will call approved packages for use.

Now, when we started working on this SCE, we were hit pretty hard by the number of things you need to consider for making a GXP compliant system. I won't go through all of these now, but suffice it to say that there is a lot to think about. And so instead of starting from scratch, we took a look at what do we already have. I mean, at the time we were able to produce all of the stuff that we needed for the reports that were going to the authorities. So instead of starting from scratch, we wanted to piggyback on what we already had.

So we wanted to launch a computer next to the one we already had. Now, the one we already had were located somewhere in Copenhagen, having a file share with all of the data and output and such. So we wanted to put it next to it, except we didn't really want to put it next to it. We wanted to put it in the cloud because we had no idea how many users were going to be on the system. There'll probably be a few in the beginning and then a lot as time progressed. And luckily we got a really, really good cloud architect to help us out on this.

And together we quickly saw that in order to fill all of these GXP requirements, we needed full infrastructure as code. And we needed to have version control on that code so that we could see who changed what and when. We also, while I quickly identified the need of machine images. So for all of our compute instances, they should be built on a machine image so that we could spin it up and be sure we would get the same result the next time as we did before. And finally, as we didn't want to do in a manual task, we wanted to use release pipelines for releasing the entire infrastructure in different environments.

So in 2020, we released our first GXP and classified environment for doing R stuff. It would look something like this where a user would come in from the workbench. Here they would have access to the RStudio IDE. They could program in R and they would have access to all of the packages they would need from the Posit package manager. They could see all of their repositories and get those from the Azure DevOps. And they could install their code on a home directory, which was a file share in the cloud.

Now we still had all of our data residing on our old computer on the file share, which were located somewhere in Copenhagen.

In terms of features, we didn't do much the first couple of years, but we did make a larger machines. Then we make more workbench machines and a load balancer in front. Because of the growing need for computation in regards to the users. And then come 2022, and we finally started to introduce new features. We introduced Python on the workbench server and PyPy on the Posit package manager. We introduced new IDEs, so that the programmer would be able to select the IDE he or she was most comfortable with.

Then we introduced a caching system. And this was because the file share located in Copenhagen had an insanely slow connection to the cloud. So this way we would only have to wait for a dataset to be transferred once, and then everybody else could use the data from the cache.

We also released a Posit Connect instance. Mostly for sharing our markdown documents and running Shiny applications. Now this was built on the same image as the workbench machine, which meant that if it worked on the workbench machine, it would almost certainly also work on the Connect instance.

But we did have one problem that was still remaining here. And that were some users, regardless of how big we tried to make these workbench machine, would hog up all of the resources that were available. So in 2023, we finally released a high performance compute cluster in the type of a Kubernetes cluster, where a user could spin up a Docker image. And through that, they would get the entire IDE, but they would be ensured that they would not step on the toes of anybody else, any of the other users.

And speaking of users, let's just take a look at the user uptake on this system. So here we have depicted the number of users that we have on the Y-axis and the timeline on the X-axis. And we can see that we didn't have that many users in the beginning as we expected. These users were mainly those that were assigned to the trials that were going to produce results that were going to use R for that. But as you can see, the time grows, we can see many, many, many more users. And in the end, we're actually not only supporting the biostatistics department, but also users from other departments within Novo Nordisk.

Package management strategy

And I thought we would have had such a great success in the user uptake on the system, if we didn't have a really great strategy on handling packages. And now, Anders will take us through the handling of these packages from CRAN.

Yeah. Thank you for that, Stephan. So the NN package management really started in 2020 with the... Remember that our organization was used to SAS and SAS only. So introducing a new language that is very much unlike SAS, that's challenging. So SAS is nice and simple. You need to install SAS 9.4. It's 10 years old. It has been patched a couple of times, but it's still in use. R on the other hand, that's not as simple. R is evolving at a higher pace. And R also has all of these packages. And each of these packages, they come with their own versions, they come at different times, they have their own life.

So these packages are what makes R great and powerful, but there is no free lunch.

So dependencies between packages, they quickly become too complex if you just pick and choose your versions arbitrarily. You need, you'll quickly create a set of packages that does not work as intended together. So we wanted to create an environment that is sufficiently fixed and stable, but still flexible. And we opted for this shared baseline model as it has now become known.

So here you lock each version of R to a particular date and time, and you install packages in versions that were available at that given date. So we do this so that we know that any pair of dependent packages, they were tested at that point in time, and they were tested that they work together. So that means that, for example, our R version 4.02, that's locked to some date in September, 2020, that we will always get dplyr 1.0.2, despite how many releases of dplyr that has been since. So teams simply just choose their version of R. They can deviate from the standard ones that we have in the system, but if they do so, they must use renv. In general, we actually recommend that you always use renv.

So what packages can and should we use? Well, they can be roughly categorized in this way. So they are the base and recommended packages. Those are the ones that ship with R. They are low risk, and they are very widely used. The contributed ones are the one we need to do a risk assessment on. So of these, there are the widely used ones. They are low risk by virtue of the enormous user base. There is the well-tested and documented ones. They may be more niche, but they are still considered low risk. It is the risk that pose a problem.

So when we do a risk assessment, we lean heavily on the R Validation Hub's recommendations on how to categorize. So the main points of categorization is the type of use, the maintenance, the popularity, and so on. So at the end of the day, this is just a human risk assessment, and the way we prompt this assessment to get made is by a Shiny app.

So our users can, in the IDE via an add-in, they will be able to select a requester app, and that simply starts up. So in the app, they can write what package they desire and the reason for it, and they can also see which R version and version of the package this will be available in. So when they submit this request, it gets version controlled in Git and gets shipped. That kicks off a pipeline and then risk metrics. The risk metric page tests or performs risk metrics for the package in each version of R that we have and that associated library, and in all of the environments that we have on the system.

Manual, sorry, a reviewer can then manually inspect these results and arrive at an informed decision. If the reviewer chooses to approve this, then it will get fed to our system code and then released upon the next release of that system.

Internal R packages

So we now have a GXP environment and we have a strategy for packages, but we still do not have R code for producing tables, figures, and listings. We do not have batch execution and logging capabilities in R, what about those? This is also handled by the packages, our internal packages, and these internal packages, they started as a single package, one package to rule them all in 2018, and this was developed on a local laptop, and it was simply a gathering of all the usual functions that have been made.

In 2020, we introduced the nnBiostat package that was simply defined as the accepted packages. So users would write library in nnBiostat and get the most useful packages. In 2021, we realized that nnR had become too big and it was split into packages related to specific tasks. So nnR then became just a simple umbrella of all of these packages, like you know from Tidyverse.

So the most important of our internal packages is nnAccess, that is for getting data without worrying about paths or data formats. It's nnPlot for creating NovoStyle plots. It's nnTable, which defines the grammar of tables for laying out tables and creating text-based output. So we wanted to mimic what we already did in SAS. It can also produce HTML outputs. We also have nnExport for exporting table figures and lists that fit into our production pipeline. And we have nnLog for batch execution and logging. Here we actually use niraspin to create nice-looking logs.

So, and although these packages, they fit together in a complicated network, each package is just like in the real world, it has its own repo. It is managed in an open-source way internally. And we have version numbers, a maintainer group, we have pipelines, and we have boards for managing bugs, features, ideas, and much more.

So we aimed to follow a common and good software principles of modularity, and we tried to build up functionality like Lego bricks. So we tried to segregate the responsibility of the packages and define appropriate extraction levels. So at the bottom here, you see the packages that are close to the system. They are system-dependent, if you will. And on top, you see the packages that are close to the task at hand. At the top, users shouldn't worry about how the data is stored, where is it stored, and so on. They should just concentrate on the task.

So for example, a programmer wishes to use nntfl to make a mean plot that will call nnAccess to fetch ADAM data through a connection set up by nnRemote. And after the plot has been made, nnExport will take care of exporting the data to an appropriate data format and associated files that we need. So this has the benefit that we can change underlying architectures in the system and simply just do small changes in the lower-level packages, if you will, and the users won't see a difference. So this is also the reason that we could inject a caching system into the system.

So let's talk about the tool chain that we use in Novo. So we use git for version control and Azure Pipelines for executing code upon code changes. We use useless for the description file and the news file. We use package down for generating a website of documentation, and we use our oxygen to write the documentation. We use lifecycle to convey the stability or the lifecycle of a function within the packages. And we use test.uncover to test the packages. We also use our markdown for writing longer format documentation in the form of vignettes. And we use DevTools to do the normal system, sorry, the normal package development.

So upon a code change, we will test again the package in a matrix way. So we test against all our libraries, all our versions of R and across all of our system environments. So we know that if, for example, an upcoming change in development to the system will break something in our packages. We then build the package and then we release it. So in the process of releasing, we are updating the package down website and we ship it to the internal repo in the package manager.

So we realized some time ago, but we implemented it this year that we need the ability to deprecate our versions. So say we no longer want to support the oldest version of R we have and stop, then we have made version specific repos so we can stop updating that. This also allows for hotfixes if system changes are needed.

So we try to ensure a high level of code quality by enforcing a main branch policy. Nobody can change the main branch directly. Only pull requests to the main branch are accepted. The pipeline needs to run with success and no errors. A feature or bot needs to be linked to the pull request so we know why we have to change. And we need at least one maintainer to approve to ensure a code review.

What R code looks like in practice

So now we both have a system and we both have the capabilities to execute R code. And I think I'll hand it over to Stefan so he can show you what it looks like in the real world.

Yes. Thank you. So I would like to show you how a script usually looks when we do it in Novo Nordisk. So all programs we have, we'll start with the header as I'm showing you now. This header will include a short description of what the program is going to do. So in this case, we're going to derive summary statistics for antibody levels grouped by treatment and gender.

We will have a very small outline just indicating the most important steps of the program, making it easier to read for another programmer. Then the actual code will almost always start with a call to library and then biostat. This will ensure that we have the packages that are most typically used in trial conduct loaded into the system and that they're loaded in the same order for every program.

And coming to the main part of the code, we will almost always use NnAccess because that's how we get access to the data. We can call NnAccess on a specific trial that will give us a trial access object. A trial access object will have access to all of the trial collections that are supported for the selected trial. In this case, that could be, for instance, ADAM or SDGM. Here we're accessing ADAM data by calling db.adam and then we can say what dataset we want to extract. And notice here that we do not have any paths here. That is taken care of by NnAccess. And this means we can actually share this code snippet without thinking about where it's actually executed. It will always get the same dataset. It also allowed us to inject this caching system so that every time a user gets a dataset that has already been loaded, we'll simply use the cache and avoid long waiting time.

So when having the data, we will usually use normal dplyr calculations in order to derive the statistics that we want to use. So for instance, here, we will filter the data. We will summarize the statistics that we want to put in the table. And then we can, in this case, group them by treatment, gender and the parameter code of interest. This will give us all of the data points that we would like to include in our table.

However, now we had this goal that we wanted to produce tables that were identical to those produced in SAS so that we could be used in our usual pipelines for producing reports to authorities. However, we were using this text-based format which has the tables that would span multiple pages and we had no support for that in R. So we created our own package in a table for doing that. And here you can initiate a table rather easily. You simply state what data is this table going to use and then you can say what columns are going to be in this table. You can do some light formatting saying that we would like mean and standard deviation to be displayed as such, or we would like minimum and maximum to be displayed with a semicolon between.

This will produce a table looking like this, but we wouldn't be quite happy yet. Now, normally we would have all of the summary statistics nested under each other. So we would have to make the table a bit better, but you can note that we have the display that we wanted and we said that we wanted in the call.

In order to nest the summary statistics underneath a new variable, we use the functionality called addTransLong where we can say we want these columns nested under a new column called sum. And we can also add some formatting, for instance, to say how many decimals do we want to apply for the different numbers. That will produce a table like this, which is a bit closer to what we usually have in our reports. However, you can see the two new columns down the bottom, the sum and the name column. Now, we would usually have the sum column nested under treatment and perhaps on the gender. And the type of summary statistics, which is the name column, would go under the parameter code.

So we could use this by having these two added lines of code where we have addTransLine where we can say summary column should be nested under treatment, which again should be nested under gender. Similar, we can add the stop of the table saying we want a name nested under the parameter code. That will produce a table like this, which looks like something we've seen before that we are going to do in Novo Nordisk.

We are able to produce other types of tables using this. So we can export this via FlexTable to, for instance, PowerPoint or a Word document or directly to HTML. But we are focusing today on what do we do when we want to export it to a simple file. And here we use the nnExport functionality from the nnExport package. Here we can simply say the output name that we want to export and a file name that we want to export it for. Then it will figure out the rest. And then we're actually done with a script that can be used to produce such a table.

And in order to run it, we will use a function called runScript that comes from nnLog. This function can take both R documents and R Markdown documents, and it can execute those in batch and in parallel. And it will give you some sort of information about how the execution actually went. In this case, we can see that the execution actually had an error. So let's inspect that log.

As you can see here, we get a nice HTML log, and that log is produced via the spin functionality from the NetApp package. We can see that we have this header on the top that just gives a short description of how did the execution actually go. Then we have the actual description and the outline that we put in top of our document. And we can see here they are nicely formatted. After that comes all of the code chunks that we have. We will also see all of the outputs in the log. And in this case, we saw that we had an error. It will also highlight both errors and warnings so that they're very easy to see as you scroll through the log.

And finally, in the end of these logs, we always have an expanded session info. Now, this session info will include both session information about the packages that we had in the session, but also packages that were simply available at the system. It will also include options that were set in the environment and other environment info. But that is actually how we execute scripts and how we do them in Novo Nordisk in general. But I'll hand it over to Anders. And Anders will tell us a bit about how we got our colleagues to actually do this.

User support and education

Yeah, thank you. Yeah, so once again, the way we have primarily been doing our user support and our education is to let everybody know that there's help to be made. So the help is here. So we have primarily been supporting people via a one-stop shop website that collects all of these package down sites. And recently we have introduced Stack Overflow for Teams as a Q&A. Until then, we have been using Microsoft Teams, we have been using DevOps, we have been using plain old mail to do this support.

In terms of education, we have promoted self-learning and learning by doing. This has been done through a podcast, as Phil mentioned. And we have internal e-learning as well. This is built on top of degree.com. We also provide lots of external references and e-learning to the outside world, the ones you can find on the web. We have done courses and demos on how to get started with R and Novo, but it's really the website that has been doing the heavy lifting.

So the website looks like this, and we call it the R.doc homepage. You simply write R-doc in the browser when you're on the corporate network and you'll be taken to this. So it's a one point of entry homepage for everything. We have the motto that everything should be here. If you cannot find it here, then it should be here and we will make it.

So this site collects all the documentation and references just mentioned. It has the podcast, it has the Pakistan website. It has vignettes, long format documentation. It has an entire output catalog so you can see how a program is structured. It has lots of guides and more. We also link further on to all of these e-learning and education sites.

Something that you all know that is present in R, this is not actually a given in other languages where you can simply write help in the IDE and get the documentation. So we, of course, also convey this to our users. And we really try to write high quality documentation on our packages. So you can get this right away. We've also tried to do the equivalent of use this in a package called nninit, and this will help you get started the first time you log in. It will help you troubleshoot if you have any problems. It will help you initialize and set up stuff, trials, repos, new packages, just in general, initiating stuff.

And we have also tried to integrate and utilize all the RStudio features with add-ins and snippets to make them a nicer user experience. The podcast is 10 to 15 minutes of how-to's of everything from the complex to the very simple. How do you get started? How do you make your first output? That sort of stuff. And again, anybody can contribute here.

The one thing I want to highlight is this Stack Overflow for Teams. We have just recently rolled this out. It's only a month old, but we can tell already now that it's going to be really good. We have already 174 questions and 255, sorry, not 55, 25 answers in there and 65 daily active users. So we really had no good place for Q&A, and we wanted to avoid answering the same questions again and again. So this site really helps spread the support burden. It helps build you a knowledge base, and it's not just for code. And this is also familiar to all programmers. So there's no education for this needed. So if we have any regrets in terms of change management, it's just that we didn't get this out sooner.

The submission journey

So now I'll hand it back to Ari, and he'll tell you about the submission.

Now, getting back to what I highlighted in the beginning, we've always believed in the power of taking steps rather than thinking about how big they should be. And to put everything in ANOS and CIFN into perspective and how you could do all of this while doing a drop project, we'll just look at a quick timeline of the development here.

So in 2019, we got the commitment to report a trial in another language than CES.