Marcin Dubel - Shiny in Action: Transforming Film Production with TARS

Hi, thank you for joining me today. I really appreciate that. I'm a staff engineer at Appsilon, the company that brought you the Rhino framework for building Shiny applications.

But today I want to share with you a story, a case study of a TARS application. I'm a delivery manager at that project and this is one of our most complex apps that we built together with one of the Hollywood film studios together.

Let me start with the question, how does the film industry work? So on a basic level you might imagine that the most important part is someone in front of the camera, actor, actress, someone behind the director and they are shooting the scenes and this is the crucial part of the moviemaking process. It might look like this or like this, so there are some visual effects happening in the background but still it's about being in front or behind the camera.

If you are a little bit more into the topic, you are probably aware that this is a really complex industry involving thousands of employees with a lot of studios, pre-production work, props creation, post-production and thousands of people involved.

However, what I learned during working with this film studio and building this and other projects together is that the most important part in the moviemaking process is the data and reports. Here is just an example of a nearly 50 pages report that they are building manually every month but they used to be building before we joined them.

The film studio's challenges

So what is exactly our case with this film studio? They have over 20 departments, you know departments like creative, music, post-production, visual effects and they are working on around 300 productions. They are reporting and collecting data about around 300 productions at the same time that are on the different stages. So they might be just the ideas or they are already in production, different stages or they are released and they keep reporting about them. Some are even abandoned but they still need to see those in their reports.

And they used to build around 60 manually generated reports mostly to coordinate the work between those departments so that one department knows that okay this is our time to take over the work and continue the process of moviemaking because the other department is ready with their work. Of course they were using some templates, we're using the previous version of the report but still this is a lot of work happening.

That's fine, okay, moviemaking requires a lot of work but then in the end they are delivering us the great pieces of cinema. But the point is that no one was happy with the result. So with those reports that they put a lot of efforts, a lot of hours into this process, they were not satisfied with the process. And this is the moment when we joined them.

And let me tell you a little bit more about the reasons behind this and how we helped them to solve them.

So with these reports there was no single source of truth. So the data was coming from different databases, sources, and you could never be sure that what you are seeing is actually up to date and this is the latest value for a particular field in your report.

There was no confidentiality and the reports were mostly shared via email and because it was a lot of manual work they kind of like tried to really put everything in a single report and shared it with everyone because that requires less working for them. However it wasn't a super efficient way to share sometimes sensitive data. For the same reason there was no customization of the content and people that were interested only in some productions they had to go through the whole report about everything to just find the information that is useful for them. Again of course they could create their reports for that particular production but one extra manual report to be added.

And there was no clarity about those dependencies between the the departments. So if you are interested in changes for a particular production you need to open this email every month, find a report, 50 pages, find this information like okay nothing changed, I can wait more. Not super user-friendly way to learn about the updates.

Introducing TARS

Okay so the tool that we created together with the film studio is called TARS and this is not an acronym. TARS is a robot from the interstellar movie that you might have seen. It's a complicated one, I don't understand the full plot but the robot was helping those astronauts to navigate, to do some heavy calculations during their travels. So this is quite accurate name for the tool that we delivered.

So what exactly it is? Right now this is a single source of all film data for all those productions that are ongoing. So it's a single place that both people that are editing the data will go there and use a really nice front end to upload the changes.

And for the data consumers this is a single place that they will come to get their data. Of course this is connecting different data sources but right now from the user perspective and for the report's perspective this is the only source of truth. This is also an automated report generation, a feature that is quite complex and I will tell you more about this also from the technical perspective how it operates. But right now all those manual reports and many more are automatically created with a given schedule and distributed for all the recipients interested.

Also this is a tool that solves the access rights. So both on a level of the application, so there might be admins of the application and regular users as usual and this is like pretty simple. You can solve that with POSIT Connect groups for example. But this is also controlling the access to specific titles and even the fields so that people will, when they enter the TARS application, they only see the data that is relevant to them and nothing else.

And this is truly a game changer for the film studio of how they are working with data and reports. So if there is any justice in this world the next Academy Awards should be something like that. But I'm not expecting that.

And this is truly a game changer for the film studio of how they are working with data and reports.

Why this matters beyond film

But why would you care? I bet that not many of you are actually from the film industry.

No one maybe. However those issues might sound really familiar with the data quality, managing access, efficient reporting and you probably face them in your organization as well.

Just briefly searching, I found tons of reports about that. This is a really common problem but this is also what we spot at Appsilon. This is a really common case. Optimizing such processes is a usual case for our clients.

I would love to tell you that you pick Shiny and this is the solution but of course it is not that simple. The project at this scale that is used across the whole organization with thousands of employees with multiple departments requires much more and the skills span across product ownership, design, software development of course.

This might be a good moment for me to appreciate the whole team both on the Appsilon side and the client side for their great work.

Shiny is a technology layer on top of all those skills required to deliver this project. And that might sound surprising during this conference but anything else could be used.

For this case and for a tool that is modifying the processes on the corporate level like this, there is no strict requirement for actually Shiny or R to be used. That project could be done with any other technology. However, my point and my takeaway message for you and if there is only one thing that you will remember of this talk, I want you to remember this that Shiny can deliver such production ready tool. And it might even not come to your mind because usually it is considered and this is what I heard from my colleagues working with other technologies that this is some data scientists can play around with Shiny to build some simple tool to show their findings in a more interactive way. But what we are proving at Appsilon for years and with this is a perfect example that this can be actually a tool on the corporate level doing anything that is required by the business.

Shiny can deliver such production ready tool. But what we are proving at Appsilon for years and with this is a perfect example that this can be actually a tool on the corporate level doing anything that is required by the business.

TARS report generation in depth

OK, so now I would like to present you a little bit more solutions in TARS that are making this really useful for our client. This is complex, this is a long project with a lot of people involved and there are some solutions that I won't get a lot into the details both on the technical side and on the user experience side. Just recently we introduced this Confluence integration for example that allows our non-coding product owner to edit the help page using the editing tool in Confluence and this is automatically and directly reflected inside the Shiny application.

Ask me later about those. What I would like to tell you more about today is this report generation that I called the TARS superpower.

And to tell you how it works I will go step by step on the report creation process and will tell you details about its capabilities and how it was implemented using POSIT tools and Shiny capabilities. So first step is to select titles for your report and this is quite interesting because in the beginning you have this step that you can select either by filter criteria or specific titles. So if you are interested only in one production, let's say the new Batman movie, you are selecting specific titles and every week you can get updates on your Batman.

However it is not the usual case. Most often our users are interested to select titles by the criteria. So for example they can pick that they are interested in the either like green so like approved titles or they can select abandoned titles and then the report automatically will select the titles for the report based on those criteria. So it might be the case that someone in the top management made a decision that the title is abandoned. They won't continue working on that and now with TARS as a report creation for abandoned titles you don't need to find this information through some different channels. It will be automatically included in your report for abandoned titles.

Okay, so the second step is to configure your report and this is to select the fields that you will include. So as you might recall I said earlier that the big problem was that the reports are super general including all information sent to all the recipients. Now you can make your report very specific. Select only the fields from a department that you are interested in reporting for your selected titles. So the reports might be really covering only that you are interested.

Okay and the first step and maybe the most important is automation of the report generation and scheduling. So how it works is that you can set your reports to be scheduled and you are picking how often it will be repeated and what is the start and end date. Similarly as you are setting the recurring event in the calendar. And with this schedule your reports will be generated and automatically distributed to your recipients.

How it works in the technical perspective. I won't get very deep but for you to understand it better.

There is an auxiliary tool called Scheduled Reports Generator. This is a fancy name for something that as you can see is an R Markdown report deployed to the Connect. And this report is set to be repeated daily. And every day what it does it loops through the list of all saved reports and checks okay which report I should generate today.

So this is like just output because this is not like a user-facing application auxiliary tool. And for the 25th of July it found four reports that should be generated that day according to what is the schedule of that report. Okay and what is happening for those reports.

The data is snapshot as it is true on this particular day when they should be generated. The data both about which titles should be included if you are using filter criteria. And what is the state of your field values on this particular day.

This data is snapshotted and saved on the other database. And this is the trick that the reports are not actually generated as a PDF or Excel files that will be really challenged to store. But we only snapshot the data that is required to generate those reports on the user demand. And from the user perspective the new report is added to their list of reports available in the TARS. And that one is generated by me so I have extra options. The other one is general that is shared by me as a recipient. So I can get this but I cannot edit, copy, or archive or even delete as I can with my POSIT.conf report. And there is also a download option. And this is the moment when the report is actually generated in the PDF form. And this is reaching out to your snapshot of the data that was true at the day when the report should be generated. And it does not matter that you are coming back after your vacation in two weeks and now you have no access to the data that was true at the moment when it should be.

The old versions are of course archived and you have only the newest one available.

Okay so this is a really powerful tool. The last step is that reports can be shared with the select recipients. Here for example I'm selecting my friend Agustin, leading developer in this project, to share the report as an editor. And he will see the same report available in his TARS and he will get email notification that the new version of report is read.

Current state and takeaways

Okay so to sum up, what is the current state? The TARS is used by hundreds of users to create custom reports to save hundreds of hours. And if next time you go to cinema, the movie will be actually created using the TARS if this is coming from that film studio.

And the thing that I wanted you to remember is that the corporate level processes can be optimized with Shiny. There are no limits here when it comes to build a really production level software. Thank you very much. You can find me here or near the Absalon booth and I will be happy to answer any questions. Thank you.

All right so we're actually right on time. So I guess the only quick question I have for you is did you meet any famous movie stars while working on this project? No, but right after the conference I'm actually going to Los Angeles, so I hope I will meet some during this visit.