Mike Cheng | The Polygons of Another World - realtime interactive rendering in R | RStudio (2022)

My name is Mike, and I'm here to talk to you today about interactive graphics in R. A little bit about me, I code in R for a living, and I believe R can be fun. What do I mean by fun? I mean things like including kittens in ggplots, running destructive physics simulations, and currently I'm experimenting with voxel rendering.

There is fun to be had in R, and today I'd like to share with you such a discovery. And that is, with careful use of base R, interactive graphics are possible, and this opens the way for new visualizations. And when I say new visualizations, I'm going to use it to play games.

There is fun to be had in R, and today I'd like to share with you such a discovery. And that is, with careful use of base R, interactive graphics are possible, and this opens the way for new visualizations.

The motivation: interactive games in R

My need for interactive graphics started with the game Wordle. This has an intuitive interface with animations and a great user experience, and so I created a clone of this in R to play. This was a text-based interface, which feels a bit clunky. It does not have the feel or the fluidity of the official game. And I realized, R gives easy access to text interfaces like Zork on the left, but I want to play actual video games with sound and color and graphics and actual interaction, like Pac-Man on the right.

So I set myself a moonshot goal. I knew where I wanted to go, but I didn't know how I was going to get there. And my moonshot was this. I will commit myself to achieving the goal before this year is out of writing and playing a video game in R.

But this then raises the question, what extra things do I need to know in R in order to get a game running? And I needed to be sure of three things, and I'm going to discuss this with you today, and that is a fast graphics device for rendering to, a fast way of drawing to this graphics device to render the game, and mouse and keyboard interaction so we can actually play the game.

Fast graphics devices

A fast graphics device is essential if I actually want to play games. I need the graphics to respond at least 30 frames per second. If not, then the games will never run smoothly. When I say graphics device, I'm referring to the area or window where the plots are displayed in R, like the plot window in RStudio.

On the Mac, there are a few other graphics devices. There's the Quartz and the X11 device. And I needed to compare them to find out what their speed and response times were. I wrote a very simple benchmark to test how fast a graphics device would respond to a simple plotting command.

And I was pleasantly surprised that the graphics devices can respond at 30 times per second. In this plot, we see the RStudio and the Quartz device both reaching 30 frames per second, with the Quartz device about twice as fast as the RStudio device. What was really surprising was to see that the X11 device can respond to commands at over 500 times per second. So yes, R does have a fast graphics device. We're over the first hurdle. All graphics devices can respond fast enough, and some graphics devices are faster than others. On the Mac, I'm going to choose to use the X11 device, because it is ridiculously fast.

Fast drawing with native rasters

Now that I'm confident that the graphics device can respond quickly, I need a fast way of drawing to that graphics device. My focus is on pixel-based games, so blocky pixels and polygons, and character sprites moving around, like the game of Pac-Man. A character sprite, like this cartoon dinosaur, is just a collection of coloured squares, and this can be drawn very quickly using just a base R graphics command, gridRect. If I draw this, clear the screen, and draw it again, over and over, I can get some animation.

To ensure this drawing system can scale, I'm going to use a common animation technique known as double buffering. In double buffering, there is one buffer for drawing, and one buffer for rendering to the screen. And I'm going to implement it in R, like this. I'm going to create a buffer in R, and then for every frame, draw the game into the buffer. And when that buffer is ready, copy that buffer into the graphics device.

For the drawing buffer, I first considered a standard R matrix. This seems like a pretty good match for representing rectangular array of pixels on the screen. But I ran into some challenges that actually slowed things down, and that is the data ordering and the colour representation within a matrix.

In a matrix, the data ordering is such that sequential values define a column. In a graphics device, it wants sequential values to define a row of pixels, so there is a mismatch between their data representation. There is also a mismatch in the colour representation. In an R matrix, it would represent colour as a string, but a graphics device actually wants four numeric values for the four bytes of red, green, blue, and alpha colour channels. So the matrix data layout does not match the graphics device, which means if I was to render a matrix to the graphics device, then every single time it actually has to do an internal data conversion, and this is going to slow things down.

There is actually a second rectangular data structure in R that is more compatible with graphics devices, and that is the native raster. The native raster is a built-in data structure in R, but it is rarely used. It's difficult to manipulate from R, but it is highly compatible with graphics devices. This table illustrates that the native raster object and the graphics device have the same data ordering and the same colour representation. This means that the native raster objects will be very fast to copy to the graphics device, as no conversion will need to be done.

So my specific double-buffered rendering technique looks like this. I will create a native raster buffer in R, then for every frame, draw the game into that buffer, and then when it's ready, copy that buffer to the graphics device, just with the grid raster command. This is a really powerful and scalable technique. It allows R to render multiple sprites very quickly.

As a demonstration that this double-buffering technique works, this is a live capture of three animated dinosaurs. This is great, but three objects aren't going to make an interesting game, and as I said, this method scales, and it scales to 100 dinosaurs, 1,000 dinosaurs, and the limit I could reach before the drawing speed dropped below the magic 30 frames per second was 5,000 dinosaurs, and I was really happy with this.

R has a fast way of drawing. If I use the double-buffering technique, using a native raster as the drawing buffer, and then copy that buffer to the graphics device when it's ready.

Event-driven interactivity

The final piece of the puzzle is interactivity, an essential element of any game that you actually want to play. A standard way you currently might get feedback from the user is to request text using the readline command, as shown here. When calling readline, R cannot do anything else. It must have the user input before it proceeds to the next line. In a similar manner, you can get the mouse position using grid locator, but again, while waiting for the user input, R cannot do anything else.

So using this style of interaction would not work for video games. Imagine a game of Pac-Man where the game stopped every time you needed to control the main character. It's not going to be an interesting game or very fun.

We have to change our programming approach and make use of the event-driven capabilities of our graphics devices. In event-driven interaction, we ask the system to monitor for events, and when they occur, we ask them to run a particular function. For those of you familiar with Shiny, it is based on an event-driven framework. If you had a button on your web page, you would not run a for loop to check whether the button has been clicked. You would ask the system, in this case Shiny, that when the button is clicked, please run my function. And that's what the observe event handler is in Shiny.

Like Shiny, our graphics devices support event-driven interaction. So we don't need to monitor has the mouse moved or has the keyboard been pressed. We can ask the graphics device to monitor for us, and when that event occurs, run a particular function. This is all handled by the GRDevice package in the setGraphicsEventHandlers function, where you can tie functions to particular events.

The events that you can watch for. Has the mouse moved? And you'll get back the coordinates of the mouse. You can monitor for the mouse button being pressed, in which case you will get back the button number. You can ask for the keyboard to be monitored, in which case you will get which key has been pressed. There is one extra event to be monitored, and that is the idle event, when no other events are happening. Consider the idle event to be like the heartbeat of an application. When the user is doing something, yes, we handle that input and change our values. But if the user is not doing anything, then we run our idle function, our heartbeat.

This means that an application skeleton might look something like this. When the mouse moves, let's update the position of the character on the screen. When the keyboard is pressed, let's run a function that will check whether the user wants to quit. And when the user is otherwise idle, render the next frame in the animation.

As proof that this event-driven interactivity works, I wrote a drum machine in R. A drum machine is just an electronic instrument that plays drums. In this drum machine interface, each row represents an instrument, and each button means to play that instrument at a certain moment in time. The events that I have tied to this application are that when the mouse button is pressed, we toggle the button underneath the mouse, and when the user is idle, we just continue to play the next beat in the sequence.

This was a lot of fun to program, and great fun to experiment with and compose my own songs. And it shows that event-driven interaction does work for what I want for games. I can simultaneously render graphics and play sounds while reacting to user input.

Porting Another World to R

I now have all the key elements that I need to play a game. The only thing left is to actually write the game itself. Now I'm not going to write a game from scratch, I'm actually going to port an existing game called Another World.

Another World is an action-adventure side-scrolling platformer released in 1991 for many platforms. It has a novel, for its time, virtual machine architecture, which makes it easy to port to new systems. It is, in effect, just a polygon engine. Rather than storing bitmaps for scenes, it just records a description of how to render that scene using polygons.

And the next scene is rendering a later scene in the game where the main character is having a drink.

So that's how an individual frame is rendered. For the actual gameplay of Another World, I created an R implementation of the virtual machine. This was based upon the JavaScript and C versions, which were originally based upon the assembly versions for MS-DOS and the Amiga. The event-driven game loop is quite simple, and I ask the graphics device when the mouse or keyboard are used, it calls a function to update the variables in my virtual machine. And when the user is otherwise idle, it just executes the next instruction, maybe to draw a polygon or play a sound.

Which means I can show you the demo of the game. This is a capture running live in R, and to preview what you're going to see, it is the typical life of a scientist. You arrive at work in your Ferrari, you drink at your desk, and then your experiment gets hit by lightning.

With that strike of lightning, you are transported to Another World. Later in the game, you are captured and you are stuck inside a jail. This 10-second demo illustrates keyboard interaction. By using the left and right arrow keys, I am going to rock the jail and escape.

That jail escape ends quite abruptly, because I have not yet completed my goal. There are still bugs to fix, and a package to release, before I can claim that I have a playable game in R.

What's next

Hopefully today with this presentation, I've managed to demonstrate that yes, with careful use of Base R, interactive graphics are possible, and this opens the way to play games and new visualizations. I have released packages to document this work, and I encourage you to try them. If you want to manipulate native raster images from R, I have released the NARA package.

I have released a friendlier wrapper around the event callbacks in the graphics devices. This hopefully makes it a little bit easier to use for games. I know that Matt Dre has used it already for his retro dungeon crawler, where we can see his blue character being chased by the yellow monster.

If you want to see an example of a complex event-driven application, I have released the drum machine as a package. And I hope soon to release the Another World package, so that everyone can play this game.

R has the capability to write and run interactive graphics in games, and I'm fascinated to see what the R community can create. Game on. Thank you.