Lovekumar Patel - Empowering Decisions: Advanced Portfolio Analysis and Management through Shiny

Good afternoon, everyone. My name is Love. And thank you so much for joining me today at Pozit Conference 2024. I work at Prokogeo. We are data and AI consulting firm based in Vancouver and Seattle, and a proud Pozit partner. I am excited to share our journey today that how we made our cutting edge tool using Shiny called Portfolio Analysis and Management, and for one of our client in financial sector. Over the next 20 minutes or so, I will walk you through how we leverage our technologies to develop data-driven investment solutions that provides real-time insights and provide them interactive tools that can make informed decision making.

So, in investment world, it's like a fortune wheel. Decision either can make a fortune or break a fortune. The timing is very essential. When you make a right decision at the right time, that can be a difference between profit and loss. So therefore, I want you to imagine a Portfolio Analyst or Manager are sitting on their desk surrounded by a bunch of spreadsheets and trying to analyze the data, spending countless number of hours to interpret the complex data. And I want you to see like if we could take all that and turning all these long hours of analyzing the data into immediate and useful insights. So that would be cool, right? So this talk is not about just speed of R, but also making an informed decision making process quickly and efficiently.

Challenges with traditional tools

So today, we will explore how R enabled us to revolutionize how portfolio being managed in investment sector can be created. So let's talk about the challenges that we are facing. So many of us know that some of the limitation of the spreadsheets and the BI tool. Those spreadsheets are very versatile, but they're notoriously error prone. Research says that 90% of the studies, 90% of the spreadsheets are containing the error. Moreover, as business get evolved, person needs to rebuild the spreadsheet, meaning that which is, again, time conserving and inefficient. On the other hand, BI tools are somewhat useful, but they often lack customization that organization need. So they also struggle with integrating with the internal data sets and face significant scalability issues.

Moreover, it also has bad approach. So for example, if portfolio managers or portfolio analysts are working on the spreadsheet by their own, so all they have different strategies to analyze the data and working with the data and implementing a method. So there is no consolidated view in the investment strategy at the organization level. So involves sharing is somewhat easier with a spreadsheet, but it often requires a validation, who did what, and while the transferring data doesn't get changed or not. And furthermore, managing in spreadsheet, the data and everything is time consuming and it's inefficient. So manual effort also requires to build reports. So these challenges significantly hinder our approaches and slow down the decision-making process and accuracy of the investment decisions.

Research says that 90% of the studies, 90% of the spreadsheets are containing the error.

The R and Shiny solution

So to address this limitation and inefficient processes, we turn our head into R and Shiny, and we develop something tool that looks like something like this, that offers the financial data, not just in numbers on the screen, but are also dynamic and interactive. Contrast to the previous approach, this approach slick dashboard provides instant feedback to the portfolio analyst and managers. And that's kind of transformation that we aim to achieve. And by harnessing the power of R and Shiny, we are not just speeding up the process, but also enhancing the accuracy and the depth of the analysis we are conducting.

So let's dive in today and explore how we can move away from those traditional approaches to the R. So we treated R as a first-class language. We know R can do so many cool things. And our entire architecture is based on the R. So Shiny provides an interactive and user-friendly interface that allows real-time insights, and it also enables users to work with the data dynamically. And the plumber API, on the other hand, it processes the backend data, and that also streamlines our business logic. We put all of our business logic and analyzing methods and custom algorithms in the plumber API, so it's separate and it can be reusable in a different part of the application.

And we use R Markdown and Quarto for automated compilable reports. And there is one product that we use that I'm very excited to share with you guys today. It's called AgGrid, which is an editable data table that allows you to edit your data entirely in the UI. And it's not just more, it has so many more features. I will just show you in the next slides.

Reactive tables and portfolio interactivity

So in this slide, as you see, I want to show you that how Shiny reactive capabilities really changing the game. We are transforming the data in real time, unlike the other Shiny apps, which has just simple inputs like a dropdown and sliders. We utilize R Handsome Table, which allows us to make entire table editable and create the Excel like user interface. And our entire table is reactive. And we utilize R6 functions and also reactive values over the reactive to make this very dynamic and allow users to create unlimited number of portfolios in the Shiny UI. And they can also see the results while they're changing the asset allocations and the data.

And moreover, this setup also performs some of the complex tasks. As you see in the screen, this asset allocation performs multiple portfolios simultaneously with ability to see how the risk and return has changed, which is one of the main statistics when we are analyzing a financial portfolio. And they can also change the allocation value from percentage to numbers dynamically. And as they change the numbers, percentage automatically changes. These are all powers of the R6 class that we utilize in the Shiny app.

And this level of interactivity is just not helping us to make a crucial decision fast, but it also helps us when financial portfolio, when portfolio analysts see something change in the market, they can quickly come here and change something and see how the portfolios are being impacted. And they can make some necessary changes.

Moreover, in this slide, so that slide was about portfolio analysts. And this is more about when portfolio managers are using this app. Portfolio managers often works with the different funds and assets, which requires thousands of scenarios. So they always conduct what-if analysis. So in typical Shiny app, it's very hard to conduct thousands of scenarios. But in this, if you convert entire data in the editable mode, we utilize the ARM or React table to make this table look nice. That often does everything on the UI side of things, meaning the JavaScript. So that's why it's very fast. And you can see that those changes quickly in the graphs and statistics table.

And we also ensure the accuracy and efficiency while we are doing this. So meaning that we utilize the try-catch and robust error handling while they are performing a data. On the right side of the screen at the bottom, you'll see some of the notifications pop up. Those are the basic some of the validation that we are performing at the end. And if anything goes wrong, we send them error. And they are portfolio analysts, and they see the errors, and they can make changes accordingly. So that ensures that we don't have to validate what they have done again. And this approach not only enhances our decision-making process, but also ensures that each decision is backed by robust validation and strategies are both dynamic and secure.

Analytical modules and data storage

And we all know R's analytical engine is very powerful. So after we construct the portfolio, we manage the portfolio, we customize it, we have different modules that can analyze it. So we have asset allocation module, we have factor analysis module that just break down the slice of data into X, Y, Z dimension and make it available to see what's going on behind your portfolios. And we utilize here the tidy finance and portfolio analytics packages in R to analyze the financial data with the precision and gaze. And we also utilize the Plotly and BSLib we just talked about. That's the second slide. We are utilizing a BSLib where all the inputs are very dynamic and interactive. And that combination also simplifies some of the complex analysis into just one format, in the one box.

After you finish up analyzing your data and making your construction of a portfolio, once you are happy with that, Shiny has a robot mechanism that can work almost every backend databases and AWS. So we store our data and analyze it, analyze data, and we also store our portfolios on the fly while they are doing it. And we store multiple portfolios into one study, and they can have thousands of studies in this portfolio. So as you can see, this can scale a lot, very heavily. And multiple users can use the app at the same time. And they all are working on the different client and different plans. So this makes our organization, investment strategy, everything into one place.

Integrating JavaScript with Shiny

And sometimes all these R packages are cool, but it lacks sometimes of user customization. And that's where we want to utilize the JavaScript. And we can enhance our application using JavaScript. And we found even use case where we cannot solve using the open source technologies. But the good news is that Shiny works really well with JavaScript. You can easily integrate JavaScript at any time into Shiny. And we build custom R package using HTML widgets to bring data visualization and data grid into R. And you can also do it very easily using HTML widgets package.

Right now, we don't have time to go into it. But if you are more interested, I would highly encourage you guys to check out the HTML widgets package, which almost most of the R packages on the data visualization are made with the HTML widgets. And it's a very robust framework to bring any JavaScript data visualization in R. And I also, if you want to take a deep dive into it, there is one book called JavaScript for R. It takes from A to Z, talk about how you can build HTML widget package and bring any JavaScript library into R.

AgGrid: editable tables in the UI

So we utilize the AgGrid in the JavaScript. AgGrid is a very powerful JavaScript library. And that allows us to make an editable table. And that is not only create an editable table, but it also has features of working millions of data in the table in real time in the UI. And you can also group the data sets. You can also perform aggregation at the row level, column level. You can also have summarized columns at the different dimensions. And there are two versions available, enterprise and community. Most of the community features have some of the features that are not available right now in some of the R packages currently. They provide them in community version. Right now, we are utilizing enterprise version because we would also like users to have creating a pivot table on the fly in the UI. Because in finance, they always use Excel. So we want to then give a same kind of feeling in Excel.

So this is a demonstration of how AgGrid looks like. So portfolios can be complex. And it can have a structure like a tree. And in this table, as you see, there are old portfolios. And there are new market values. As you change the column on the left side, it sums up on the right side. It also collapse multiple rows. It can filter down based on the values, what values are. And it works really well with the thousands of rows. Everything is fast because everything happens on the front end side. And this functionality of the collapsible column calculated field custom filtering, which greatly enhances our user experience. And this module we call the portfolio rebalancing modules.

Plumber API architecture and logging

And after AgGrid, I want to talk about all these cool things about UI. But we also created plumber APIs for Empower back end. So our entire business logic effectively separated from the user interface, which makes our app very fast. And this separation also streamlines the business processes and makes sure our core business remains maintainable and reusable across the different applications. Plumber API also allows us to perform asynchronous programming using future package and promises. And it also has an option of the caching mechanism, which further advances your app. So this architecture not only improves our productivity, but also ensures that our system can handle large volumes of a dataset with ease.

So I want to show you this diagram where we have a database. We have data virtualization tool that just prepares and cleans our data. And we bring everything in the plumber API. And we run some of the custom algorithms. We also run some of the, using some portfolio analytics packages, we create some of the, and analyze and visualize the data. And we directly render in the Shiny app. And this data analyzed can be consumed by Shiny app, or it can be consumed by any person who utilize R, or can be utilized by a person who using a Python. So that is the main advantage, one of the thinking where we move to the plumber API. We move our all business logic in the plumber API.

And this plumber API also has some of the logging feature as well. So in financial industry, accountability and transparency are very crucial. So, and we want to ensure that all actions taken care by portfolio analysts or managers are being logged. So we utilize logger package and you can find the link on my website. I have a detailed article, how you can implement a logging mechanism in your Shiny app and plumber. And this logs features, logs of every user action, providing a detailed record, who did what and when. And this is essential for auditing purposes. And particularly when you're reviewing the investment strategies.

In financial industry, accountability and transparency are very crucial. This logs features, logs of every user action, providing a detailed record, who did what and when.

Real-world impact and conclusion

And what it is, all this work that we have done has some real world impact. The implementation of our tool has transformative impact on our clients organization. Firstly, it has significantly increased the decision making efficiency and providing a real time data analysis and accurate insights. Users can now make informed decisions very quickly and move away from the traditional tools. Secondly, this interactive and user friendly nature of the tool is very important. User friendly nature of the tool and how's the user engagement and everybody walks together in the fun place. And this analysis process make more enjoyable and productive. Finally, the scalability of the customization option ensure that our tool can grow because we leverage the Shiny modules in a way that each separate components has their own part. And we also have R packages that can process some of the data pipeline and that help us to make change quickly when we need to.

In conclusion, our journey of building portfolio analysis and management tool, our Shiny and plumber API has been both challenging but rewarding too. We have successfully addressed the limitation of the traditional tools and providing a solution that offers a real time insights and interactivity and scalability. This tool empowers the managers to make decisions better and ultimately enhancing the organization efficiency and effectiveness. I hope this presentation provided valuable insights for you how R and Shiny plumber can revolutionize the portfolio analysis and management.

And thank you so much for your attention and I'm ready for any questions that you guys have.

Q&A

Thank you so much, Love. We have a few questions from the virtual audience. There's some interest in your methods for handling multiple users making input changes in the same portfolio at the same time. Yes. So we developed asynchronous programming in our approach and we also track the session ID as long as as soon as the user starts an app. Every user has a unique session ID. So we utilize that session ID to make sense of what users are doing, what changes. And we utilize on the both sides. We have a session. We also make sure that session ID is similar when they access the plumber API and when they develop a Shiny app. That's how we leverage who's using what and what not.

Interesting. This may also have some applications to this next question. But what tools have you used to manage user auth and user data access? Yeah. So we right now we are utilizing RStudio Connect and RStudio Connect has some of the classes that you can available to see who's logging in and who's not. And right now we are logging at the person who deployed an app. But we in that log we also putting the name of the person who's using an app. Logs being generated by person who deploy an app by default on RStudio Connect. But we utilize the session authentication provided by RStudio Connect and we put that into our log.

Excellent. OK. So any interest in putting that JavaScript editable table into an R package? Yes. So that's very interesting question. And I am right now working with my internal team to make that package available on the CRAN. And we are working hard to make sure that at least community version is out there. Excellent. I'm sure there's going to be a lot of interest in that.

So is the source data also dynamic in the in the situations you showed in your presentation? So yes. So not in my presentation but we have one module called market dashboard where users analyze the data what's happening in the market. And we utilize a reactive pull mechanism and we utilize the memcached under the hood which is a type of database or live database in the user session. And we that market dashboard basically process the data every 15 minutes and put it in our module. I have not enough time to go through it but we do have that and we utilize the reactive pull function in the base R Shiny to make that happen.

That sounds very cool and it looks very intuitive but it sounds like you made some complex method into looking very intuitive. So congratulations on that. I have one more question here. Were there times that you thought maybe Shiny wasn't the right tool and you needed to go more traditionally full stack like Angular, JavaScript or C Sharp API work? Yeah so that is a very cool question. That was one of the debates when we started this project. And we wanted our portfolio analysis was somewhat familiar with R and we wanted to leverage R as you know because they can easily understand and we can also work with the stakeholders and domain experts at the same time. And we and I'm pretty sure R has a week so we try to do with the some of the stuff in the Vue and C Sharp but we went back and we we come back to Shiny. The main reason is that I think we want to bring everybody together into one place. So you can think like portfolio analysts using R, managers use mostly like Excel or BI tools and some of the asset managers they want some reports that can be generated by them. So we tell them we develop in Quarto and we have some of the front-end developers who work in the AG Grid site. So we also that we all of us can have one release cycle that help us improve our productivity in operation sides really really well and that is the reason we chose this architecture.

Awesome, great work Love. Thank you so much for presenting today.