Mehran Moghtadai @ TD Bank Group | Data Science Hangout

Transcript#

This transcript was generated automatically and may contain errors.

Hi everybody. Happy New Year, if I haven't said it yet to you, and welcome back to the Data Science Hangout. I'm Rachel. I lead Customer Marketing at Posit. This is our first Hangout of 2024. The Hangout is our open space to hear what's going on in the world of data across different industries, chat about data science leadership, and connect with others facing similar things as you. We get together here every Thursday at the same time, same place.

Is it anybody's first Hangout? Say hi in the chat, as we'd love to welcome you in and say hello to anybody joining for the first time too. We're all dedicated to keeping this a friendly and welcoming space for everyone, so we love hearing from you no matter your years of experience, titles, industry, or languages that you work in. It's also totally okay if you just want to listen in here, and awesome to be a part of the party that happens in the Zoom chat too.

But there's also three ways you can jump in and ask questions today or provide your own perspective on certain topics too. So the three ways to jump in are you can raise your hand on Zoom, and I'll call on you. You can put questions in the Zoom chat, and feel free to just put a little star next to it if it's something you want me to read out loud instead. And then we also have a Slido link where you can ask questions anonymously, and our team will share that in the chat here in just a second.

A quick note that if you are watching the recording at some point in the future on YouTube and want to join us live, the link to add it to your calendar will be in the details below to join us. And there's no rule that you have to stay the whole time or talk. You can come and go as if it's your schedule. But with all that, welcome. I am so excited to be joined by my co-host for today, Mehran, AI and ML Accelerator and Enablement at TD Bank Group.

Because after a while, we started having this relationship with the business where they knew that, you know, what we gave them, if it wasn't perfect, they could come back to us and we would take it a step further, right? And having that sort of like trust, that's what kind of enables a lot of these projects to kind of get off the ground.

Sustaining innovation: the handoff challenge

Yeah, sure. Hi, Moran. I'm really curious. I've seen tension in the past with like accelerator or innovation teams making really cool things and then running into problems of like, how do those get supported long term? Because like these little innovation teams usually aren't resourced to handle that. Do you run into that kind of tension? And how have you figured out how to like make things that are sustainable and know when to hand them off and know that you've got partners who can take them?

Yeah, for sure. So this is a very important point. And I think that's been one of the biggest challenges that we've solved sort of incrementally. So one of the most important things is even before starting a use case like this. And I think, you know, it requires a certain level of maturity that I would say like early on in our journey, we maybe didn't have, but we were kind of lucky in certain ways, is you need to be very direct with the business and make them understand that like, look, we can build this, we can build this very quickly. And it's fun to do to build, but it will require maintenance and maintenance means there is costs associated to that. So we need to make sure that whatever they're asking for has benefits, right, like actual tangible benefits, so that when we do go back to them and say, like, look, I need to, you know, assign like an FTE to kind of do long term support for this. They're okay with funding that, right.

And, you know, we need to be very direct about that very upfront, right, which is why a lot of the use cases that we tend to support, they're fairly high value. The other part of it is, you know, making things sort of standardized. And when I say standardized, I don't necessarily mean that, you know, make like a cookie cutter application that kind of applies everywhere, but have these sort of like frameworks that, you know, you customize that, you know, if you take somebody that doesn't have experience in this, but he's worked on a similar application, he can come and he can support it as well. So by doing that, we're able to have like a fairly small group of people that can be the sort of ongoing support team as well as the development team.

Then there's another part of the equation is that, you know, you start kind of looking like an IT team once you start building all these layers, and we're not an IT team, right? So then you need to have a proper relationship with your IT team and kind of, you know, set the limits of like, okay, well, the infrastructure you own, you own sort of like, let's say even the deployment of the applications, but like the L2 support, like if there's something wrong with the application and stuff, then that's what we own. So making sure that you have these operating models that kind of make this work, that's sort of like the magic recipe, because if it's all on the, let's say the analytics team side, that's not really going to be sustainable and work long-term. And if it's all on the IT side, you know, we've seen that that doesn't work either. So having this collaboration is really the key.

Team structure at TD Bank

So, TD Bank has a lot of different lines of businesses. So, you have these various lines of businesses that each have their own sort of, like, leadership. All of the enterprise data and analytics is essentially under a single leadership, and we have different what we call, like, practices under these leaderships. So, you know, we have a BI practice, we have a sort of, like, a data engineering practice, and we have a AIML practice. So, we reside in the AIML practice, but I would say we're very well integrated. So, we have very close connections with the line of businesses. Let's say, like, all the lines of businesses, the ML, AI practitioners, they're all grouped under one, let's say, VP, but we all work very closely with our, you know, respective lines of businesses.

Do you, like, get together with, like, community initiatives or, like, annual conferences or things like that, too?

Yeah, for sure. So, we have town halls, and again, depending on, you know, we have, like, three different town halls depending on who is included. So, we have our own sort of, like, the AIML practice, and then we have the larger enterprise data and analytics family, and then you have the different town halls with the lines of businesses. And I mean, you know, the enterprise data and analytics community at TD is very large. I would say probably we're talking about, you know, 4,000 or so data practitioners. So, this includes the AIML practitioners and everything.

Open source adoption in a regulated environment

Yeah, absolutely. I work for Charles Schwab. So, I feel like we're, like, cousins or something. One of the challenges that I face here is kind of an aversion to open source packages, products, anything that's not kind of vetted at enterprise, purely from, like, a risk perspective. I've had so many conversations about, you know, even well-vetted packages like dplyr or pandas, and, you know, how can we guarantee that there's no malicious code that's going to, you know, come scrape and steal our clients. Have you run into that? And how have you, like, maybe kind of put those fears aside and kind of, you know, confirm that these are really safe and well-vetted tools in a variety of industries?

Yeah, that's a good question, because I would say, so, 10 years ago when I started, I remember. I started working, and I started using R because I kind of used it for my thesis, and, like, very quickly people are, like, wait, what are you using? Like, we use SaaS here. We don't use R. Like, this is open source. Like, how do we know it's, like, the right outputs and everything? And I was very lucky because my manager actually, he told me, like, don't listen to them. You do you. You know, you're doing great work. But as I would say, like, very quickly within, like, the next few years, I think people started seeing, just in general, like, different pockets starting to adopt these things because it was solving a lot of issues that, you know, like, SaaS was, like, way behind.

And I think naturally we kind of, like, went through this direction where things started getting accepted, and I think in the meantime what was happening is that there was some leadership changes that were happening, like, enterprise data and analytics team was being created, and we were very lucky, and to this day we're very blessed that, you know, our leadership, they're very forward-thinking. Like, to them, it's a non-issue, and even to this day, like, for our enterprise data and analytics team, we're really a code-first sort of mentality where, you know, for example, one of the big projects that we have right now is creating our own in-house machine learning end-to-end library, right? So this is, like, a big project, and we're building this ourselves, and it's using, like, a fully open-source stack.

TD also acquired an AI company. I was one of the first banks, at least in Canada, to do so, and then after that, like, other banks started doing that as well a bit. So, I would say we've been very lucky that we kind of organically managed to adopt a lot of these tools, and we had the right leadership in the right places to kind of, you know, allow that. With that said, it's not always without challenge. So, for example, you know, we still have challenges bringing, let's say, third-party models, because it's like, well, what do we know about these models, right? So those challenges still exist, and we have partners in our Enterprise Protect team that actually, you know, they proxy all the libraries, so we don't have direct access to the libraries. All these libraries get scanned before they get put on our internal repository.

So, we do have these layers of control that are kind of put in place, and, you know, again, the other layer of control that we have is any model and stuff that we create, right? We can create models with anything that we want, but if we want that model to go into production, it's not like we can take any package or anything that we want, right? Everything does get vetted by a model validation team. So, that model validation team is like, okay, like, this library that you're using, they look into the documentation, they look at all the methodologies that are used, and they end up vetting this. This can be a very long process for certain models, and, you know, it's grueling, and it's not the funnest thing, but these are the controls that they've essentially had to put in place so that we can do these things in a regulated environment, right?

Working with security and data sensitivity

Yeah, for sure. So I think it's very important to have like a good understanding of their framework. So I think every larger company will probably have their own sort of, you know, layers of control, right? So there is privacy partners on one side, like they tell you, you know, this is the data I want to use. This is what you're allowed to do with the data. And then you have sort of like the frameworks that are sort of like set by whoever is their sort of like security team, right? So in our case, you know, the way that it works is that, okay, well, you have this zone. In this zone, you're allowed to do X, Y, Z. And this is sort of like the guardrails that are put in place.

The difficulty happens if you have like a use case that doesn't really fit those patterns. And that's where it becomes challenging. Because when we were migrating our Posit Connect platform onto Azure, they were like, well, the Azure environment isn't really made to have data coming into it in this kind of fashion, right? So that, again, was not an easy conversation, because they had to understand exactly what is going on, they had to build new patterns. But once we got to the point that, you know, the patterns were built, all the conversations were had, Enterprise Protect gave their sign off, now you have this pattern that you have that you can reuse and kind of do whatever use case that you need to, right?

So I think the key here is to be able to like abstract your use case into a pattern that you can kind of explain to the partners and say, like, this is what I need to do. And you also have to be kind of like, you know, open at the end of the day that they say, like, okay, well, you know, to make your data secure, you need to do X, Y, Z, and kind of like hear them out as well. So that it's not easy. And especially if I take the consultant point of view, I feel like that's particularly difficult, because you know, you you're kind of like time bound. With that said, the benefit that you have on the consultant side, I think is that, you know, if you're paid by the company, they want to see results. So I would say use your business partners to kind of like, you know, push some of those conversations along with the security partners.

Because a lot of the conversations that we've had, you know, I can see is like, if I go to them with like a use case, and I say, hey, I need to do this, and it requires some new patterns, they're gonna be like, no, we're not going to do this. And essentially, if I look at it from their perspective, like, why am I going to bother creating some potential risk for this thing that you're coming to me, right? So you do need to have that support from the business side that says, like, no, this is important. This is going to enable lots of value for us. And we need your help to tell us like how to do this, but do it within whatever like risk appetite that you have. Once you have that support, it becomes a lot easier to kind of push things along.

Sandboxes and synthetic data

Yeah, sure. So I'm at Prudential and a similar scenario where we like to hear from external partners on how they do things and how they solve issues. And there's there's two things that we use very successfully. One is sandboxes. They're, they're kind of the environment that is in production. But it's completely separate from the corporate network. And there's just nothing you can do to break things. And another thing that we were very good at is making fake data that looks a lot like our data. And then, you know, ask the consultant or the external party to develop a solution that works within that.

Yeah, that's a that's a really great point. So the sandbox is very key. And this is actually, I think, a concept that, at least for our IT department was a little bit foreign, I would say, like maybe going back like a few years, because to them, it was like, well, you have a dev environment, and then you have like all these like, you know, SIP, PAT, whatever you want to call it. But what we eventually convinced them is that, well, we kind of need a dev, but we need a dev that's with like real data, like essentially a sandbox, because working as a data scientist with fake data becomes a little bit difficult, right, when you're actually trying to prove results and things. So, so yeah, so the sandbox idea of like, you know, it, it looks like a production, it breathes like a production, but it's not production. That's something that I think is key for any data science team to be, you know, to be useful, right?

The model validation team

So the model validation team, very important team, very popular in terms of like anybody that is doing any model, they interact with them. So that team, I would say the, the background of the members, they're typically PhD, very highly skilled in, you know, the academic side of model. Like they kind of understand really like, you know, all like the splitting methods, all like the little detailed methods of, you know, how do you treat your data and what impacts those things can have on your model. That's really their, their skill set.

I would say they have a particularly difficult job as well, because first of all, like, you know, everybody's doing models, right? So you have a lot of models coming from different lines of businesses and you can't really look at a model in a vacuum. You need to understand sort of like the business context of the models. And if you have like a centralized team that kind of deals with a lot of different lines of businesses, that means like very quickly learning about those lines of businesses and that particular use case. So they, they also need that skill set of like, you know, being able to talk to the business, understand what is going on, and then also have a very deep understanding on the technical side of the, of the model.

So, so yeah, they're very, very important partner and they're, they're really key on getting, so that's, that's, you know, one layer. And then we have a different layer that is more on the bias and fairness side. So getting the model validated is more of a technical question of like making sure that the model is going to produce the outputs that you expect and everything. But then after that, it's like, well, all these variables and all these things that we used in our models, is it even fair to use those in the context that you're using them, right? So, and that's like a different partner that we, that we work with.

Career advice and team vision

Mehran, a question I love to ask everybody here is, what is one of the most memorable pieces of career advice that you've either received or given throughout your professional journey?

Yeah, that's a good question. So one of the things that, you know, so my, my manager for a long time at TD, so, you know, I worked very closely with him and like, I wouldn't say it's necessarily an advice that he gave me, but it's something that I really learned from him, like working with him and being in meetings with him is, you know, like, if your vision and your goals are set in, you know, in the right way, right? So, and when I say that, you know, pessimistically, you know, you can look at your career and say like, well, you know, I want this job, so I'm going to do whatever it takes to do that. But I think the better way to kind of look at your career is like, well, what do I expect to kind of like achieve, right? And what are sort of like the vision that I have for this company, right? And like, think big and what are the steps that you need to do to kind of achieve that?

Like, if you have that kind of a mindset and you do everything, you know, in that kind of direction, you will achieve great things. And not only that, but you'll band together like a group of people, right? And you have a certain motivation because all you're doing isn't like, well, you know, this model is going to bring this benefit, let's just do it and get across the line. It's like, okay, well, you know, this, for example, this call center app, it wasn't that, you know, we wanted to do this and like solve the problem. It was like, we realized like, oh my God, like people are on the hold, you know, it's during COVID for like an hour and like, we need to help them out, right? So having that kind of mindset that, you know, at the end of the day, like it's the clients that matter and what's my vision and how do I think my skill sets are going to, you know, help that out. That's sort of like what brought us our team to this point.

Like, if you have that kind of a mindset and you do everything, you know, in that kind of direction, you will achieve great things. And not only that, but you'll band together like a group of people, right? And you have a certain motivation because all you're doing isn't like, well, you know, this model is going to bring this benefit, let's just do it and get across the line.

I love that. Thank you so much. So do you give your team an opportunity to like meet, to talk about those big picture things or where do people usually?

Yeah, for sure. So we do, we try to do twice a year, what we call an offsite. So we kind of get together. We don't talk about necessarily sort of like the specifics, but we talk about sort of the division of the team and, you know, sort of some of the past learnings, what went well, what didn't go well, what we want to achieve, like sort of in the coming year. We also kind of combine that with some like activities, just, you know, some team building things. Like last time we were, we kind of got everybody in Montreal and we actually did like an art, like a little art workshop together. So I think that kind of thing is important because a lot of these sort of like discussions about like vision and like about the company and things like that kind of arise from there.

And TD does this annual, and I think they're going to make it more frequent, but what they call TD Pulse, which is they sort of get a pulse of everybody. And we try to talk about those things like very openly and try to, you know, make things better, not just like in terms of our work, but for ourselves, right? Like to be able to produce good work, everybody needs to be happy at the end of the day.