Eric Wanjau & Ian Muchiri | A touch of R in Robotics | RStudio (2022)

Hello, and welcome to our presentation. We hope you've had a great conference so far.

So there's always a thing that goes like this, all is well that starts with a good story. Sorry, I made that up, do not search for a saying. However, my colleague and I thought it would be great to tell something that resonates with us, our experience, and our talk. So here's a quote from one of our favorite books, Alice's Adventures in Wonderland, and it goes like this. For you see, some of the weird things that happened lately that Alice had begun to think that very few things indeed were really impossible.

So while we're talking about this, I'll ask to put it into a little bit of context. Please take a moment and watch the following clip.

As you can see, this is a voice control robotic arm, which is starting to go to certain locations on a chessboard and do some tasks. So what if we told you the majority of this was done in R?

My name is Eric Wanjau, and I'm joined by my lovely colleague, Ian Muchiri, and today we're talking about a touch of R in robotics.

The story behind the project

So you may be wondering what would make two guys do a project in R, in particular robotics. So here's the story. One of the reasons that led us into this adventure was hardly made home. So a while ago, when Ian and I were getting started in R and doing some of our electrical and electronics engineering projects, we reached out to him asking him whether R can program some of these devices we're dealing with.

And also a good recommendation for some machine learning books. So hardly being hardly was really honest with us, and he told us that at that time we don't know anything about Raspberry Pi or ArcGwin or some of these devices, but it really gave us a good starting point in machine learning. So thanks for that, hardly.

And with this response, it got us thinking, so if hardly doesn't know about this, we figured it out and shared it with the R community, then that would be a really great impact to the community in general. So this is one of the reasons.

So the second reason was that Ian and I were involved in California Data Science Club in our former university, Data and Commodity University of Technology in Kenya. And we kept on getting these questions. So we see a lot of ingenious things happening with R in other domains, such as sports or medicine or pharma. What can R do in electrical electronics engineering?

There are two reasons that really made us think and, you know, made us try out R in what we love doing, and that is electronics and robotics and all that stuff. So this is where the story began. Anyhow, back to R and robotics.

So in this talk, we'll be presenting a series of steps in which we used R to achieve motion of the robotic arm as you saw. And hopefully at the end of it all, you'll see how R played a pivotal role in the success of this project.

Setting up the hardware

So Ian, to move a simple robotic arm, what do we actually need?

Yeah, to move a simple robotic arm, the first thing that you need to do is actually set up your robotic arm, which basically means setting up the hardware that you're going to be using. And the beauty about this is that the pieces already come, you know, pre-cut. All you have to do is actually remove the pieces and actually assemble them together. And once you do this, it only takes about 30 minutes to an hour. And with the right enthusiasm, you can actually do this. And you're already done with the first step, which is actually setting up the manipulator.

Voice input and transcription

So after you're done with the setup, after you're done with the hardware, you proceed to ask the next logical question becomes, what are the manipulator inputs? And just as a disclaimer, in the course of this talk, I will be using the term manipulator and robotic arm to mean one and the same thing. So don't get lost in that.

So we realized from the video that Eric just showed us that our manipulator used voice as an input. So how did we achieve this in R?

So the first thing that we did was actually record the voice using audio package in R, whereby we recorded the command B2 and G2. Once we have our voice input, we transcribed this using Azure, Azure Cognitive Services, and transcribed the voice input into text using HDR package and JSON Lite package. I'm sure some of you are familiar with that.

After doing that, the transcription and getting the output, we proceeded next to do some assertions to ensure that indeed the format, that result that we got from Azure is actually correct, because you realize in some cases there are some errors. And after doing this, we extracted the characters from the text, whereby B2 was the initial position and G2 was the final position.

So to make the API call to Azure, we used the sample code shown in the slide, whereby all we needed to do is actually define our subscription key, which we actually got from Azure once we set up our cognitive services on the platform, define the context type, which was audio file, the flavor of English that we were using, which was US English, upload our recorded audio, got the transcribed data using post request to the API. And then from this, we actually pulled the initial position and the final position in text format.

So just to recap, what's the state of data at this point? So at this point, we had our voice input, B2 and G2, ran it through Azure, and then we got a string equivalent, a string which is B2 and G2 representing a specific chess box actually, which is B2 and G2, which is the final position.

Defining the workspace and coordinates

So after deriving the string, the next bit becomes defining the workspace whereby the manipulator will be working on. And to answer this, we ask ourselves the question, where is the manipulator actually going? And I believe, Eric, you're going to tell us where was the manipulator going?

To answer your question, Ian, what we need to do is to assign a special coordinate to the chessboard location that we just obtained from the transcription. So how did we approach this?

So we first started by creating a virtual board in R using ggplot. Probably a lot of you have used it to create some elegant graphics in R. Once we did this, we extracted the central pixel location. And then the next point was translating the pixel location to actual real-world board coordinates in centimeters. And to do this, we used the scales package in R.

So why did we use this approach? So we figured out that different boards come in different flavors and dimensions. So we needed a virtual board that is static. And then depending on the dimensions of the chessboard, one can then translate the pixel location to actual real-world location depending on the dimensions of his or her board.

So at this point, we can combine everything and put it in a good old table where we have the position of the square box on the chessboard. And most importantly, the x and y coordinates in centimeters that would represent a location of a chess piece on a chessboard.

So let's wrap this up. So we started with speech and then we got a transcription. And then for each start and end point, we now have x, y, and z coordinates that represent the position of a chess piece on a chessboard. So for instance, B2 can be represented by 6 centimeter on the x, 15 centimeter on the y, and 3.5 on the z. So 3.5 is just the height of the chess piece that will be moving around on the chessboard.

Inverse kinematics

So the next logical question now becomes, how will the manipulator get there?

Yeah, so to answer this question, you just take a moment and just observe the video that's playing. And you realize from the video that these three blue things that have some cables connected to them, those are actually motors, and they are responsible for the manipulator motion. And in this case, we had three motors which were controlling motion. And yeah, and you realize with a motor, the motion of a motor is actually rotational in nature. And from Eric's point, we now have our coordinates in terms of x, y, and z. So we have to find a way of linking the x, y, z and the motor angles. Converting x, y, z coordinates into the equivalent motor angles. And to do this, we use a branch of math called inverse kinematics. I know it's a mouthful, but just bear with us for a second.

So at the moment, let's treat inverse kinematics as a black box. So what inverse kinematics does in a nutshell is, it takes x, y, z inputs, converts them into the equivalent angles. Therefore, we have our x, y, z coordinates, which are actually converted into motor angles that can now move the manipulator.

And just to give you a feel of what inverse kinematics is all about, I give you some fair warning. There's some math ahead. So just hold on tight and just bear with us.

So to do inverse kinematics, it's actually based on trigonometry functions, which are the things that most of us learned in elementary and just need to brush up a little bit on. And the beauty about this is that most robotic arms, once you buy the kit, it comes with the instructions to inverse kinematics. Most probably people, there are people, there are ripples out there whereby people have already done the inverse kinematics, so you don't really have to know, you know, the deep stuff about these things. So you just need a high level overview and just use the code that is being used by other people.

And in case it's not there, in case it's not provided, you can actually come up with your own. And this is a sample of what we actually did. And as you can see, we use the basic, you know, tangent sine and cosine functions that, you know, we learned in elementary. So all you need to do is actually, is actually, you know, brush up on those skills. And then, yeah, we'll probably give a more detailed approach as to how we came up with these equations in our repo.

So just to recap, what's the state of data at this point? So at this point we got our audio file, which was the input. We transcribed it and got the string B2 and G2. We actually transposed the B2 and G2 into the real world XYZ coordinates that represented, you know, chess piece locations of the chess board. And using inverse kinematics, we're able now to translate the XYZ into the equivalent motor angles. And you realize that since we have three motors, the way we identify which is which motor is actually by using the string.

So A, letter A corresponded to the first motor, letter B to the second motor, letter C to the third motor. So 112 A meant 112 degrees to motor A, 126 B meant 126 degrees to motor B, 107 C meant 107 C, 107 degrees to motor C.

Moving the robotic arm

And finally we go to the fun part, which is actually moving the robotic arm. So Eric, how did you move the manipulator to the desired positions?

So let's answer Ian's question. At this point, we have everything that we need to move the chess pieces on the chess board. So you know the location of the chess piece, and you know the angles required to move the manipulator to that position. So we now need to find a way of linking R and the manipulator and how to do this. So this is the architecture that we used, where R is doing all the heavy lifting and finally sending a string of data that Ian just described. And at the middle, we have an electrode device known as a microcontroller. And a microcontroller, you can think of it as an electrical device where you can connect multiple peripherals, such as motors or LEDs or speakers or sensors, then define some instructions on how to send electrical signals to manipulate those peripherals that are connected to it. In our case, these are motors. We want the motors to rotate to some given angles. So that was the work of the microcontroller, which in our case was Arduino, to just receive a string of data and then send a signal to each respective motor. And then the motors would rotate and bring the manipulator to the desired position.

So once we had this figured out, the next logical thing was now to ask ourselves, how do we send data from R to the microcontroller?

At this point is when we realized that most microcontrollers can be programmed via a serial interface. So a serial interface is a communication interface in which data can be sent and received sequentially. So with this, we just had figured out how to set up a serial interface in R. And then a microcontroller can read that serial interface and send an electrical signal to move our motors and bring the manipulator to the desired position.

So how do we set up a serial interface in R? Surprisingly enough, there's already a packet that helps us do this. So the packet is known as packet serial. So with serial, the serial packet just requires a few parameters to set up the serial interface. And some of these parameters, important arguments rather is the port number. So this is the USB port, which you connect the microcontroller to your PC. The next argument is the baud rate. So this is the rate at which you send and receive data, which was 9,600 bits per second. So you want a speed that is just right, not too fast and not too slow. And the next parameters are there by default, and they just help in structuring how data is sent to the serial interface. For instance, do you want a new line appended at each instance of the data? What character do you want to append after each new line current return? So this just describes how to structure data that is sent to the serial interface.

And after you've done how to set up the serial interface, you now write data to it. And to write data to it, we just combined the strings of angles into a vector. Then we use path work to write each element of the vector to the serial connection. So once the data is on a serial connection, now it's time for the microcontroller to read that data and do its thing. So these are the instructions that we put on the microcontroller whereby it checks whether data is available on the serial, reads data from the serial, passes integers to extract the numeric part of the string. And then depending on the case, say A, B, or C, it writes that value or that integer to the respective motor associated with a string. For instance, if it sees A, it knows I'm sending an angle to motor A. B sends an angle to motor B.

And at this point, we have achieved the basis of our whole project. We started from transcribing data all the way to knowing the angles to which to move our robotic arm. And finally, moving our robotic arm using the help of a microcontroller. So with this, you can set up some, I would say some more complex tasks, such as achieving a capture move using our manipulator, like we're going to see in the following clip.

So this is basically using all that information that we have, you know, that we have described, and just adding some layers of complexity to it. And at this point, we hope you have successfully showed you some touch of our robotics.

Pros, cons, and what's next

So everything has some pros and cons, right Ian?

Yeah, definitely. Everything has its own pros and cons. And in the course of this project, we realized that there are some key areas that we wanted to highlight that are really excelled. And one of the key areas that we saw that are excelled was when it came to data manipulation, it was easier to do data manipulation with R because I realized that R is actually optimized for such tasks. And also we realized in the course of the project that a big chunk of robotics is actually a data problem and it can also easily be solved in R. Therefore it became quite easy using R.

And also R is actually less computationally expensive as compared to what we've used before, which was MATLAB. It's a software that actually is used by guys in our field, specifically the guys doing research. And the reason why we say this is because we realized with MATLAB, once you boot it up, it loads all the packages that come with MATLAB. But we realized with R, you only load the packages that you need. Therefore it becomes a bit light on the PC and it becomes easier to use. And also R is open source, therefore meaning anybody can actually reproduce what you did once they have access to the R. And actually it becomes easier to actually do this without having so much of resources.

And also we realized in the course of the project that a big chunk of robotics is actually a data problem and it can also easily be solved in R.

So the main issues that we faced when using R was that the serial communication, which Eric just highlighted, was a bit slower. Yeah, and we thought that maybe this could be an issue with the package and it's something that we are willing to look at in future and maybe offer some suggestions. And also the audio package that we used to record audio was a bit, it wasn't up to date because each and every time after recording an audio object, we had to restart R so that you can actually record another audio object, meaning you can't do two consecutive inputs without first necessarily having to restart R, which was a bit annoying and, you know, it was a bit hectic.

So what's next? Because, you know, like, what's next? Which is our next adventure? So currently we plan on adding and contributing R packages, specifically packages geared towards the engineering field, because we believe that given our background, this is something that we can actually, you know, add to the community and actually build the community. And also speaking about communities, we also want to build R communities specifically in engineering, such as what we have in, you know, medicine, R in pharma. We want also to have a community of R users that are actually in the engineering field. And also because we are adventurous, we hope that we'll explore the unexpected uses of R. And once we are able to get something, we'll share with everyone.

And Eric, I don't know if there's anything else that I left out. Not much, Ian. Everything was really well said. And thanks a lot for that. So hoping that we get to the next adventure soon. So, but anyhow, we really wanted to wrap everything up, but just noting that a lot of the majority we did, we did using packages that are already in the R ecosystem. We thought when we were starting this, actually we had to reinvent some tools, but there was no need for that. Everything we used was everything that has already been built and we just had to modify it to our needs.

So we hope we really present a good case of an unexpected using R. And mostly we hope that we've inspired one or two to try out, you know, R in their domain or, you know, R in some fun activities they have, you know, just using R everywhere, anywhere.

So anyhow, time to wrap up. So we'd hope to wrap up with, you know, let's say a quote from Alice in Wonderland, like we did from at the beginning, but we did not find one that's really befitting to our story. So we made one up. So if there's ever a story written about Eric and Ian, we hope it goes like this. You know, for you see, somewhere out of the way things had happened lately that Eric and Ian began to think that very few things indeed were really impassable and an emphasis to the R in impassable.

So thank you so much for listening to us. Please do have a good conference ahead and take care of yourselves. So talk soon and bye. Bye.