Aleksander Dietrichson | Mobile Education App during COVID | Teaching Data Science | Posit (2022)

Thank you and my co-presenter as well John is here. He will not be presenting because he can't be bothered I guess. Is this better? Yeah he works for the Department of Ed and we work together in... John works for the Department of Education and we work together in Chi-Square Lab. So that's about us.

So I'm gonna set the context and by the way this talk isn't actually... I teach data science but this talk is not about teaching data science. It's about using data science to teach. That's sort of the essence of what we do in learning analytics. We collect data from educational interactions and then we analyze that and try to provide feedback to instructors.

Context: COVID and emergency online teaching

And the story here is as we all know there was a pandemic in Argentina. We went into full lockdown in March of 2020 and then that meant that all instructions needed to be online in a hurry. Since we're in the southern hemisphere this actually coincided pretty much with the beginning of the semester so we didn't have to do it mid semester. But what happened is that everyone started to teach using Zoom or any kind of video conferencing system that was available and we found that less than half the students actually had access to a laptop. So in practice they were going to follow the course on a device, on an Android phone typically.

And so what we wanted to do, and this is something that we worked on before, is to help them create, help the instructors create a feedback loop. Because as Kelly mentioned in her talk there is a lot of nonverbal communication that goes on when you actually meet face-to-face. That's part of the reason we're all here. And you lose that, you lose some of it at least when you do video conferencing. Especially if the class is a big class with a hundred students or something like that. So what we set out to do was to mitigate some of this by creating a tool to to provide feedback so the students could provide feedback to the instructor.

And we did three iterations of this. One was slapped together very quickly because we were in a hurry in I think less than a week. And then we did, called that the emergency alpha. And then we did a proper alpha version in April. And then we did what I would call a beta version in August of 2021. So for the start of that semester.

First iteration: Shiny Mobile as a PWA

And the first one we did in Shiny using a package called Shiny Mobile. And then we hooked it up to a backend and we we have a package called Cognito R that allows for password protection of Shiny apps that we used. It's available on CRAN if you want to use it. And then we distributed that as a PWA. And this is what it looks like. So essentially the two, on the left there, the two icons are just links to two Shiny apps. One for the instructor and one for the student. And they have a login and that's handled by by Amazon. And then we have a really easy questionnaire with some some questions. One of them was about connectivity because it turned out to be important in this context. And then there's two open-ended questions there. You know, what did you learn in this class and what would you like to to have reviewed? And this goes back and gets analyzed and we use R for that. And then we provide feedback to the instructor on, you know, the student feedback is summarized and provided back to the instructor.

And this is the same package, Shiny Mobile. It provides interface items that look good on a cell phone. So and this is a completely, you know, completely reasonable way of putting a Shiny app on on a cell phone. It works, it updates very quickly. It's very lightweight. It's only a link and it's cross-platform. It works on an iPhone, it works on on anything that can show a web page. We did find though some bandwidth issues because if everyone tries to log on to, we published this to shiny.io, if everyone tries to log on at the same time to provide feedback, it actually crashes the server. Now you can solve that by, you know, by throwing hardware at it, but that's not really scalable. So and you know, I was also a little uncomfortable that this is not a native app. You can't send people to the Play Store or App Store to get it.

Second iteration: Android wrapper

So the second version, we use the templates from the Android community and there's several of them floating around. And there's one that we have that we used that you can see there on the slide. It's essentially a wrapper that displays a web page. But this one goes on the Play Store and, you know, you need to know a little bit of Java. And if you hate Python, you probably hate Java as well. But it's like two lines, so it's not that bad. And it's still just a Shiny app in a wrapper, so it's still not really scalable. And it will only work for Android, which is a slight drawback.

So this is a slide from one of the instructors who used the app. And this is what it looks like. This is probably a later iteration. And so she used this to show her students the feedback that she was getting from them and show them that this is actually useful and, you know, what did we learn. And as you can see, you know, it's summarized in not a word cloud format, because that wouldn't really look good on a device. So we used some other tools there.

Research results

And we got research results from this, which show they were quite encouraging. And so the scale here is 0 to 10. That's how it's graded in Argentina. Anything below, so 4 is the lowest passing grade. So that part of the scale isn't actually being used. And so, you know, the distributions are not normal. And there were some outliers like you could see. So we, you know, we use candles for correlations. And then we use ordinal regression instead of linear regression. And we found during simulation that we could actually predict. This is post-hoc though, but we had predictive power. After just three weeks, we could pretty much tell which students were going to finish the course.

After just three weeks, we could pretty much tell which students were going to finish the course.

And we got good feedback from the instructors. They said, you know, this is actually useful for us. There are some issues. Some of them were actually mentioned in the last talk. There's a lot of linguistic models that you can get access to, but they are typically either too general or too specific. When you're teaching class, and the example here is class in linguistics, introductory linguistics, it's often very specific vocabulary that you have to learn. And that will always sort of mess up the statistical models that have been trained on, you know, a billion words. Because this is, you know, these are words that you use in two papers in your life. And then you don't repeat them. And then there is the issue of Spanish versus English that we actually saw some of in the last post. We did most of this in Spanish, obviously, but we are also working on an English language version of the same. And also, you know, the modeling tools and the tools for doing NLP tend to be anglo-centric. So that became a bit of an issue as well. But I think we resolved most of them.

Beta version: Flutter and plumber API

And then we got to the beta version, the one that's out now. And that's a true native app in a framework called Flutter. The programming language is called Dart. So we've sort of, we've moved the backend to the Google Cloud Firebase system. And now authentication is done by Google. And we have all the analysis done by R with a plumber API for, you know, so that the cell phone will get its information from. And then we have enhanced content recommendations. Because one of the features of this system is that if I'm teaching statistics and no one got the standard error, I may repeat that in class. But if, you know, two people did not get what a standard deviation is, I may not want to waste class time on that. And I can send them over to, you know, watch a video or something. And we try to automate some of that, which is interesting for large classes. And then we did an email interface as well.

By the way, this is what it looks like now. You sign up with a QR code, which is convenient. If you're teaching online, you can show the code on the screen and they can copy it right away. It's a little more awkward if you're actually watching it on a cell phone. But, you know, it's also, you sign on with one link and it's authenticated by Google and all. And then we did, we used Blastula to send out an email report to the instructor. And this was also, you know, sort of a way to get over what I perceive to be dashboard fatigue. That people don't actually go and log into the dashboards. They, you know, but if you get it pushed into your email, maybe you actually consumed information.

But that turned out not to be the case. Once we did this, we got this feedback from one of the instructors. And Cecilia and I are married. So this was not, this was a request that was difficult to ignore. And in fact, it turned out to be impossible. So we were sort of full circle. We came back to Shiny again, hoping that at least we have the instructors on Shiny, which would be a small, smaller percentage. So perhaps this is still scalable. It would be a lot worse if the instructor and 200 students would have to access the server at the same time.

So this is an example from her course. And this is also an example from the same dashboard. So these are some content recommendations that the system came up with for the course that I've sort of assigned this to these specific students. That's an artificial intelligence piece that we are working on. And that's what I had for you. This is our contact info. We're happy to answer questions, both of us. So thank you.