Model Monitors and Alerting at Scale with RStudio Connect | Adam Austin, Socure
Deploying a predictive model signals the end of one long journey and the start of another. Monitoring model performance is crucial for ensuring the long-term quality of your work.
A monitor should provide insights into model inputs and output scores, and should send alerts when something goes wrong. However, as the number of deployed models increases or your customer base grows, the maintenance of your monitor portfolio becomes daunting. In this talk we’ll explore a solution for orchestrating monitor deployment and maintenance using RStudio Connect. I will show how applications of R Markdown, Shiny, and Plumber can unburden data scientists of time-consuming report upkeep by empowering end-users to deploy, update, and own their monitors.
Timestamps:
2:01 - Start of presentation
3:43 - About Socure
6:12 - Model performance matters, deployment isn't the end of the story
7:26 - What is a monitor?
8:55 - What is an alert?
11:00 - Monitor example
18:09 - Firing an alert from RStudio Connect
19:00 - Why monitor from RStudio Connect?
24:33 - How monitoring drives success at Socure
30:00 - Git-backed deployment in RStudio Connect
36:00 - Shiny app that their account managers see
46:00 - Architecture of a monitoring system
56:00 - Connect hot tip System-wide packages
57:00 - Why did we try Connect for monitoring
59:02 - Why do we keep using it for that :)
Resources shared:
Blastula package: https://github.com/rstudio/blastula
connectapi package: https://github.com/rstudio/connectapi
rsconnect package: https://rstudio.github.io/rsconnect/
Intro to APIs blog post: https://www.rstudio.com/blog/creating-apis-for-data-science-with-plumber/
Speaker Bio:
Adam Austin is a senior data scientist and RStudio administrator at Socure, a leading provider of identity verification and fraud prevention services. His work focuses on data science enablement through tools, automation, and reporting
blastula
connectapi
plumber
rsconnect
rstudio
Shiny
