Simon is a longstanding contributor to Django and a member of the Steering Council and the Security Team. We discuss his current work at Zapier, type hints, django-ninja vs Django REST Framework, and thoughts on the future of Django.
Carlton Gibson 0:06
Hi, welcome to another episode of Django Chat, a podcast on the Django web framework. I'm Carlton Gibson joined as ever by Will Vincent. Hello, Will!
Will Vincent 0:12
Hi, Carlton.
Carlton Gibson 0:14
Hello Will. Today we've got Simon Charette who's a longtime contributor to the ORM. And all time hero of mine, Simon, thank you for coming on the show.
Simon Charette 0:21
Yeah, thanks for having me content. Well.
Carlton Gibson 0:25
Brilliant. So for me, like, okay, so I'm like flush to to have you on the show. But you're one of the big contributors to the ORM. And nobody knows who you are. So before we kick off, could we tell us who you are and how you found Jiang and what's your backstory? And how have you come to be in the community?
Simon Charette 0:45
Yeah, sure. So when I, I was in uni, I discovered Python by there was this very pesky HTML thing that was popping up when I was trying to connect to the Wi Fi and wasn't paying to kind of open my browser. And you know, like it, oh, if you're allergic to like an HTTPS site, it wouldn't work. So I figured out that there was a way I was using Ubuntu at that point. And it was a way to use one of the signal emitted by the OS to I guess, like book into it, and in Python was the way to do it. So that's that start. That's how I start using Python. Just trying to avoid having to fight with my wife about university Wi Fi. Portal.
Carlton Gibson 1:28
Fantastic. I love that story.
Will Vincent 1:32
Well, it sounds like you've maybe already done a little bit with programming languages before.
Simon Charette 1:36
Yeah, correct. So my, my dad was working for a company called Citi Group, Diners Club, it's a kind of like, credit card thing. And he had an office in Montreal, and a couple of time per month, you needed to like work on the weekend, I guess you'd like backups of like credit card stuff. So you had like to print everything? Or like the credit card or the training data paper for others.
Carlton Gibson 2:02
Who was the on the paper with the little dots down the side? Yeah, exactly. Yeah. Okay, wow, boxes
Simon Charette 2:07
and boxes of that. And so I and there was like, crazy server room there as well. And I got to kind of play with computers at a very early age got to program kind of like using like Visual Basic. So like writing, bash, and what's the name of the windows equivalent? Like DOS, I don't remember all the, like, I was writing like bat files and things like that. So So I started playing with that there was I play video games after that, and trying to automate a few things that maybe I shouldn't have. But like I, I was kind of like obsessed with these games and wanted to like it turned into I like more the programming aspect of playing the game itself. So So yeah, I did like a background in programming before using Python, I work in the web agency developing websites in PHP, and maybe my first contact with or am light stuff is that we had this in house. PHP framework, like a lot of agency at the board. We had a database connector, logic that was mainly talking to MySQL. And we had a customer that wanted to use thing was access was or SQL Server I don't remember exactly, but I had to write a kind of like connector for the ads kind of like pour to the logic that was dealing with kind of like pagination and all of that stuff. So yeah, that was kind of like before Python was doing mainly PHP and there was a framework as well that was popular before jQuery called prototype Jas and the community was not that open but I kind of like I contributed to certain extent was reporting back to it. But and I remember going in that camping trip with my parents and kind of like being obsessed with this thing and just printing the source code and the favor just carrying it with me because I wanted to read it all. So yeah, that was there was all kind of like pre Python stuff. I did PHP and JavaScript I guess. Yeah,
Carlton Gibson 4:22
cool. Cool. Cool. I've got this vision Have you been like the girl from Jurassic Park being like this is a Unix system I know this
Simon Charette 4:29
I did. I was more kind of like REO level before like I didn't I didn't add like a lot of the focus on the kind of like the US things at that point. But yeah, I like computer stuff. And I was exposed to it at the early age because of the gym. I might that had Yeah, and
Carlton Gibson 4:51
like the giver what the tell there is that you were already using a bunch of when you went to university and so
Will Vincent 4:56
yeah, that's a trip trip me up. I was like, No, that's not a and I I think that's for people learning program. That's the, the thing is that some people come in with years of experience and some people, it's their very first programming class. And so it's sort of like if you can get through that first year of programming, most people can catch up. But you don't know that someone has, you know, in no other subject does somebody have 510 years of experience, but in programming that can be the case. Yeah,
Simon Charette 5:23
that this experience kind of like oriented my let's see, carry choice as well. So when I got to college and uni, I mean, I was I didn't know what to do, really. There was tons of things I liked. And but I had kind of like this, this pool towards it, because it was interesting to me. And I think it was rewarding as well, because I'd say what was easier, but I had like, adversity, doing these things. And that was in college. I went to a college called Kenosha Mizell. Nov in my was working in my, my degree was in multimedia. So basically, we're doing kind of like drawing of like 2d animations, 3d animations, and a little bit of programming. But since I had like vestiti, in programming, it was I was very interesting, because I was able to do focus on the other things basically, that I eventually didn't end up working with, but kind of like gave me a foundation in kind of like, some form of like video editing. And like, the part of the part that I liked the most was really about just throwing, I was not that good, but kind of like being able to spend some time running at school whisking fleet, I was not expecting that when I was in high school that you could do that
Carlton Gibson 6:44
is something nice about GM, those kinds of the other bits which aren't so you don't, you don't think of them as your core bits. But those other bits that sort of round you out a bit, like make you a bit more of a fully fleshed human being. Yeah,
Simon Charette 6:59
yeah, it was. Yeah, it was, it was good. It was a great time, like being able to just explore that was kind of like the the coach part. And yeah, it was really cool. Okay,
Carlton Gibson 7:10
and how did you find Django?
Simon Charette 7:13
So, the agency where I was working at was the added consultant. And one of the recommendation that consulted gave them is that in order to speed up their development process, they should adopt a framework because like a lot of agencies that were maintaining the old PHP framework. And the consultant was someone coming from a Python and Django shop, and you was kind of like trying to push for that. But they wanted a external assessment. So I was there. And I was kind of like, this BHB guy didn't know much about Python, except for the script that I wrote for Wi Fi stuff at the uni. And they Yeah, they asked me to kind of like do an assessment because they had a trust relationship with me. And we evaluated both symphony and Django and we decided to do a complete 20 D moving from PHP sharp and going to towards Python, and Django. And yeah, that's how I ended up using it. I think my first contribution was a translation that was not properly done in I think that was even like before trans effects. But I ended up kind of like creating a ticket for it and started contributing to a few more things issue that we ran into. And eventually cloud powers wrote to neon clothed, wrote to me on on IRC and asked me if I, if I wanted to help with contributing a bit more, because that was kind of like submitting something every weekend. I guess that was always work at that point, you just have to, to be to be so pesky that the they wanted you and so you could help instead of bugging them.
Carlton Gibson 8:56
You just have to keep turning up, right? Yeah. This guy game, right. Okay.
Will Vincent 9:04
Well, it's interesting hearing that story about how even when you were younger with prototype, Jas, like it didn't, it didn't occur to you not to reach out, you know, because I think a lot of people think frameworks or software is, you know, people over there, do that, and I do this here. But as you found it an early age, right, if you show up, it's especially open source, like people are dying for help. So it's that that leap that was a big one for people I find, you know, later in life, right, because they have imposter syndrome and all these things, as opposed to just doing it incrementally. And then. So that's a smooth transition from you know, being like, Oh, here's this new thing and up, something's wrong, like, of course, I'll fix it. I think that's still something people don't think about or, or don't know where to look or don't don't think that it could really help them. And
Simon Charette 9:51
I think the Django community and the Python in general is, is a good example. It's like trying to do things right and being welcoming Um, in terms of trying to provide guidance being open, that's something that I think made me for fall in love into Jing with Django is that but my experience with real time GIS it was kind of like a wall. It felt like I, I couldn't. I was suggesting something and it was kind of like there was no kind of like no discussions in I get that Django is is this a large project as well as possible that you cover sometimes run into more of maybe a push back against and some things but I still do feel my I tried to embody that to a certain extent try to be as welcome Ming as possible, while maybe maintaining some some boundaries as well, because we're humans, and we don't want to burn out of yesterday.
Will Vincent 10:44
So did the so did the entire consultancy then switch for all its projects from PHP symphony to Django? Or is it just on that specific one,
Simon Charette 10:54
it was a prototype project. So we wanted to take care of it now is was a great that it was it was kind of like matric base that we had like two or three developers that we knew that get get similar projects, using our framework. And there was like a new engineer as well or in there. Like it took a lot of time to onboard that the people on the previous PHP projects, we wanted to compare that. And even with the fact that with the language barrier, basically, they needed to write the learning new language and deployment as well at that time was not as easy as it is. Now that was kind of like, maybe 15 years ago. Now. I don't know, like, it was a long time ago. And we still ended up being kind of like flush, while learning all of these things along the way. So then a few more projects were added. And they ended up switching entirely obviously needed to maintain the PHP projects. But the new projects were created using Python, the thing
Carlton Gibson 11:55
that you said a couple of times during the introduction during that story, there was the they will make like everybody else at the time everyone was maintaining their own in house framework. And I think that for me is that is the big one is like why? Why use a framework? Well, because if you don't, you're going to end up inventing one, and the one you invent just won't be as it won't be as good. Yeah.
Simon Charette 12:17
And they all kind of like onboarding was kind of like spoke to me a lot during that time, because like you don't, don't have to maintain like incomplete documentation. You don't have to keep up with like the latest security. Victor's discovered a lot of like, the batteries include their bike as like a peace of mind that allows you to focus on not being in the business of maintaining this kind of project. So yeah, it's a big plus. And I think a lot of people realize that, that that time, kind of like the rise of Symfony rails Django, kind of like it was that time when folks start to realize it. Yeah, there's there's a benefit in kind of like putting all that together. Yeah,
Will Vincent 13:02
exactly. Did you did you poke around rails at the time as well? Because that was also the you know, the other big one, did you Did that come up, or no,
Simon Charette 13:11
that's during the review there. When we got like, compared to technology, I played a bit with it. But I, the part I did the most was kind of reverse engineering, but kind of like looking at how they were doing some things just because I was curious. Because, I mean, it's kind of like let's say a sibling of Django to a certain extent, right. The grew along over the years. So it's always interesting to see how they're doing things. And now they're, they're taking the project. But no, I've never capsulate chosen it and use it for a project and myself.
Carlton Gibson 13:44
So I wanted to talk to you about your work because your work exactly right. We should be a big company and a big Django shop. And can you tell us about your life there?
Simon Charette 13:55
Yep. So I previously prior to Django, I was working at a store I found a startup that was yet another email marketing platform that was focused EU laws that were coming into effect in Canada about kind of like spam laws with regards to explicit consent that you need to renew when sending email marketing and company were eager to be able to just comply and make sure the do anything they shouldn't. And there's also this dynamic in in Quebec and kind of like maybe more to eastern part of Canada around just bilingualism to be able to send email campaigns in both languages, tools like MailChimp and such to do a bit of that, but it was not. We were kind of like trying to target a niche there and focus on these kind of like two big features that users in Canada and most seemed like a Quebec quarter year to get. So I worked there for five years. And that's kind of like the time I started computing more and more to Django because we were doing things with kind of like dynamic models, in graphic all sorts of crazy things that needed me to dig into, like what the RM was doing, or like bugging NCAA contributer at the time about, oh, well, this thing is, you have an idea.
Carlton Gibson 15:30
But yeah, so Alright, so this is the secret is basically you got to be because you'll you'll want, you're basically the main or one of the main contributors on the ORM for the last few years. And your depth of knowledge is just astounding. So the real trick there is that you're trying to you're trying to push it to its limits unit for a long period of time, and you slowly creep the those that knowledge that but
Simon Charette 15:55
I got to get into this as well, just a relational database, like every time I tried to look at them or read about them, I find it fascinating. The way they implement these kind of like low level primitives that can be used to do like, nowadays, tools like Buzz dress, and you can use it for so many things when you start. There's everything in there that relates to kind of like that, or synchronism. Across, like distributed service. So yeah, it's fascinating to me. And it was also fascinating to be dig in there and see the discussion around, like the expedition APIs that were those added the work that was done as well for ants by NC by to try to just make things better. That was very interesting to me. And I had the chance to be exposed to all sort of like these discussions, because I was invited to Django Under the Hood for, I think, the three years in Amsterdam. And that was kind of like, very kind of like, I felt like I needed to give back because everything that was kind of like, given to me there in terms of I mean, it completely changed my life. I was kind of like included in this group of, of folks that were leaving, I'm just indirectly mentors, there's there's an extend kind of like being able to wash out the work, the approach problems, and just maybe get things done in in a context where it's hard sometimes to drive consensus, and there might not be like a right answer. There might be some stuff are better than others. And at the end of the day, we need to make a choice. So yeah, that was, that was the thing that pushed me towards the ORM trying as well, maybe to fill a bit of the void in this area. Because it's kind of like an important part of the framework. But it's, if there's no kind of like, keeper of the knowledge there is easy to make you repeat errors of the past tests can only do so much in terms of telling you to not do a particular thing.
Carlton Gibson 18:09
Yes, yeah, no, exactly. So I guess that leads in? How do we, if something he talks about in your Django con keynote in 2022? But how do we how do we encourage contributions to the ORM? Because this is it is quite big and scary. And it is quite complicated. And it is quite difficult. But how do we? How do we keep it so that there can be new people coming on board to pick pick up and pass the baton on to and keep it fresh?
Simon Charette 18:38
I think that we need to put out with some guardrails, I think the code base is once once you wrap your head around it. There are obviously parts that are more intimidating than others, for example, things that relate to join pruning and things like that these are kind of like, non trivial to reason about, but most of the time, you don't have to go that deep. There's two layers and the could be better documented, something that I like, of you three things that I mentioned, the at the end of the talk was a form of like mentoring. I was I've been kind of like looking at the site about what like the projects like Django nuts are doing trying to maybe find a way to contribute there and chime in, find a way to mentor developers that might be interested to dig into that. So that's on my couch like 2024 To Do List, let's see if I ever get to it. But other things we could do that I was against at first because of the burden mind pose but any form of like typing as well in terms of like self documentation. I mean, I guess it would help because if you've added like your ID and tools are so good nowadays in terms of like generating codes or like guiding you to a certain extent that you can get 90% of the We there. And that's enough to kind of like just start wrapping your head around all of these pieces. And obviously documentation, but that's a large effort. And it's, it's there's a tax with documentation if we need to like to keep it up to date. It's one more barrier to kind of like, trying to change things up in this area. Yeah.
Carlton Gibson 20:21
So you said you said a couple of things in in one you said earlier on dynamic models. Let's come back to that in a second. You just said talk about timing is Do you see that a way of incrementally introducing typing into Django? Yep.
Simon Charette 20:35
As I think that in, like the past releases, we've, we started to maybe like the past for these fibers, we start to kind of like document change to the database. backends change, even if, like, if they're not documented, we're still pointing like, Oh, if you use this particular method, you might want to do that. So that might that might be one area where you start typing it. Maybe trying to incorporate things from project like dang, Django stubs could be one. I mean, a lot of the work has been done there. To me the the immediate benefit in terms of like meeting DRM too, would be more around adding it at the lower level. So more journals inside inside yet, so we can add these kind of like play with it see how much of a burden it is setting up CI pipeline as well. A in my buyer provides like Alex for other solution for forcing typing products also of bells and whistles to kind of like do it in an incremental way because it was developed to the point where there was pre existing code base that didn't have typing. So from my experience on other projects at Zapier, for example, it works pretty well like the gradual typing. Sure, there are new things that you need to learn about. Typing can be intimidating, partially in the case of like Django, where we do kind of like some meta class magic, in some cases, you have to, it's not as easy but I think there there are real benefits there in terms of self documenting code.
Will Vincent 22:11
Well, let me ask you a question. So Zapier for I think, seven years, what is what kind of projects do you work on? So you've mentioned that they're very supportive of you work doing open source work? But is it like one big project for multiple years do you have different ones like what is what is the day to day look like?
Simon Charette 22:30
I would say it's kind of at this point is more like multi month project where I am more kind of like try to support many teams in terms of features or trying to deliver enduring problems they run into, there might be a couple of times where I asked to write a piece of code because it's, I'm I have knowledge about many systems, and just getting the teams together to do it would be art. So sometimes I just kind of like create proof of concept or actually deliver, let's say that like I impact code that allows for down blocking of teams or prevented like lifting dependencies between between two teams. So otherwise, a lot of mentoring, a lot of trying to come up with the way to drive alignment within the company, around the technical vision is something I've been working on in the past few months. So best thing best in July, when I joined, that was my title was product engineer. And I just kind of like revolved around the workflow, part of the product, which is like very interesting, again, it's kind of like the sad equivalent of the RM for for zap here. It's just fascinating. This gets like huge distributed system that you can just throw a workflow definition at it. And you see, you want to trigger off something, and it's just, it does its thing. And it's resilient, and in the face of like, the 1000s and 1000s, of API's that can return very various response code, and you need to make sure that some things are either important. And like, it's, it's very interesting from a software engineering perspective. So
Will Vincent 24:16
it sounds a little bit like a software architect would be like a title I've heard for what you're doing and that you're not directly managing people, but you're overseeing large groups of engineers and providing guidance and occasionally coding, but largely helping them figure it out on their own. Is that accurate?
Simon Charette 24:34
It's uh, yeah, so it's a good definition. I think it's a it's about that. I think it's a I wish I could call more and I think that Django is the Catholic escape from that. But I realized that the moral code that like and others as well as they try to make it explicit to me because I, I like thank you, obviously, but I can have way more impact by empowering engineers developers with 10s App. You're to actually ship gold themselves and Catholic get a vector of that it's, it's, you know, like the the the writing code and shipping it and seeing it deployed like ultimate this machine the reward cycle is so immediate that it's it's yeah, it's easy to get drafted there while you can you can be way more impactful doing other things.
Will Vincent 25:23
Well, that sounds a lot like. So Andrew Godwin has also had most recently had a software architect title and it seems like that's about as good as it gets as you progress in a company because companies want you to manage manage people. But if you want to still code a little bit and have impact, but not be doing as much managing managing that software architect role is the way to go. But then you need Yeah, it's almost like you need open source just to code code or code for fun because you're removed from it. It's a it's a weird thing, right? Like, there's no other me because I didn't grew up programming, there's most fields, you just do what you're doing, and you get better at it, and you just do it, but coding, and I guess most engineering fields, very quickly, three, five years in, you're just really pushed to provide guidance rather than do the work yourself. So it's an interesting, interesting dynamic, again, not coming from a programming background growing up. But sounds like you're managing it well.
Simon Charette 26:22
Yeah, it's a it's, at first, it was hard. It sounds like, like explicitly extract myself from there. But Catholics, as I said that the rewards the reward cycle is not immediate, it's kind of like longer now. But when you see these things happen, even if you just have like this in direct impact on, you know, like putting this little cog in the soul delivery thing, it's it's so much rewarding to, to see that you, you've kind of you've you've held still to a certain extent, because that's the thing that I've used your warning from even for Django just feels like no, you out to a certain extent, someone with the problems they're facing, and just see the impact of it as well. So yeah, it's different reward cycle, but still very rewarding. But
Carlton Gibson 27:10
you do have quite a lot of availability for Django, it seems like you know, when there's an issue, you're, you're always, you're always there to help feel that. So Zapier must be supportive of your contribution. And yeah,
Simon Charette 27:24
the feedback I've had was basically as long as it doesn't affect my, my kind of like performance at work. It's okay. And so yeah, I do I liked I like the Django stuff. I like the aura thing. So I gravitate towards that. And sometimes it's just kind of like just a nice break of some days or problems. I'm dealing with that job. I'm working, I think organizational. So that's like going back to Django is, it's just a nice way to kind of like clear my clear my mind and just maybe put things in perspective as well.
Carlton Gibson 27:55
I mean, I said them from Dan's point of view, there must be two sides to it. One is that obviously, your knowledge VRM is an asset to, you know, the ability to maintain the product and your ability to maintain the products as a team. But there must be downtime and gets a mental health break as well by contributing, as
Will Vincent 28:14
well. I'd say number three is also Simon, anyone who works with Jango enough is going to come come across you and it's another it's a recruiting tool, right? Like, I mean, I would think that'd be a big draw to work with you. If anyone's interested in Django. Right, I would fire was a manager. If I was a company owner, recruiting is always the problem. So I think it's hard to quantify, which is sort of the issue. How do you quantify someone's time, you know, on the steering Council or the Security Council, but you know, if I was looking to apply for Django work, if someone was was, you know, very involved, that would be a huge draw.
Simon Charette 28:50
It's yeah, I think there's a Yeah, obviously, there are Catholic benefits for ZIP pros fu as well. So I think that's one of the reasons why they are motivating. They're motivated by that but it's it's great to have this flexibility it's works well for me. So I'm I'm glad but yeah, you're right. Like it's I am able to open house for sometimes non trivial questions about all things should be done, or we have like a very large mono repo. Like a very, like old Django project, like created like No, like 12 years ago now. And it uses like a huge MySQL cluster and tons of other databases. So there are stuff that come up that we need to come up with solutions that are non trivial, and adding some knowledge about DRM certainly helps. Okay, so
Will Vincent 29:45
I think Zapier is primarily MySQL is that correct?
Simon Charette 29:50
The project that was created a long time ago, that it's kind of like the sky like the bubble it is MySQL and Postgres. So we are Uh, we, we use it for like all the historical models are there. And all the new ones were kind of like trying to break the the monolith kind of like as each domain use a separate database, so we can kind of like maybe stuck as broke to a certain extent. Because you start to run into issues with Team boundaries. The Gulf like domain violations that Django kind of like allows you to basically create a foreign key across like, everywhere. I saw I heard you call to the previous Django chat talking about the old kind of like, user slash user profile thing, as well as something we can't like go went back and forth about because the project was created, before custom user model was supported. So we've got this model than when Django switched to adding customers or model, it felt like it, that's ferocity. But the more I think of it, I think it's great that it's, like separated. So I, let's say that I support you in this, this. This front, I feel like it's their pros and cons, basically. And because these models can grow very large. That's it. There's pros
Carlton Gibson 31:15
and con, there are trade offs. And it's not like, it's clearly the case that you know, this, but in my experience smaller, many, much smaller scale projects and Zapier, but it becomes a dumping ground, it becomes lacs, throw another field on the user model. And then that's fetched every request. It's like no, this isn't it, we're not encouraging a good pattern here, I think is the bottom line. And so there may be some projects where it costs but you know, don't get rid of the ability to swap it by all means, but as the recommended path I just added. And again, it's, as you say, trade shows, but it's so it's so
Will Vincent 31:52
well documented this I mean, so I'm giving a talk in two days at Django, Boston, largely just cribbing Carlton's ideas. So I'm gonna give Carlton credit about the Django user model. And I mean, 12 years ago, Russell, Keith McGee put up a whole wiki. With the five it's more than five options when they were talking about what to do, there was like one to A to B to C, and even then, you know, just pros and cons. And, you know, I think Jacob at the time was sort of in favor of option five, which is more what Carlton is going for with with just like, just basically just an identifier. But you know, there had to be a B DFL benevolent dictator for life decision. And they chose, I guess, the custom user model approach. So it's just fascinating that it's not anything new. And it's documented, they had to make a decision. And
Carlton Gibson 32:40
in Russell's Uriarra, he gave her a Django con about that when it was introduced
Will Vincent 32:44
to a user. Yeah, it's a great
Carlton Gibson 32:46
talk, and you're nodding all the way through and you're like, Yeah, that's right. And I think it's but you know, like this, coming back to them, like, let's change it anyway. Anyway, that's a thing. Simon, I wanted to ask you about, oh, unless you've got if you've got something to say that.
Simon Charette 33:04
Oh, yeah, I was just, I think it's a good idea, swipe a little model, I think it's great. I mean, like users I've been asking that we lean in to that a bit more. I think the migration framework is really, to a certain extent, Andrew spent, and all the contributors around the migration stuff as well spent a lot of time kind of like making that better reuse. So I think there, there's value in kind of like, providing abstract base class. And in most cases, like adding a single user model is going to be fine, because you're not going to end up with like 50 fields that that makes, like the tuples that database so large that fetching them for every request is going to be problematic. So I think it makes sense. And I wish we're leaning more of it towards swipeable models with a lot that people, not everyone loves them, but I think they're clever idea.
Carlton Gibson 33:54
One thing I wanted to ask you is talking about migrations is I have like a kind of concept of sort of bolt on migrations, where it kind of an optional one. So I've got this package that just makes the use email field unique on us it unusable. And all it does is it comes along on the app installs, it adds a constraint and then it's got a migration that you run one time and it just all works smoothly. And if you wanted to uninstall it, you just reverse the migration and, you know, pretended it was never there. Do you think it would be feasible to do a bit more like a bit more along that? I mean, it's a kind of a horrible monkey patch, just a proof of concept that you can do login by email very simply with the with the default model. But do you think there's a possibility of kind of optional migrations that third party packages can provide say, Look, if you want this do that? I
Simon Charette 34:48
know there's a way I guess you could have like different app conflicts that possibly so we've we've not done that extensively, but I think the idea when Emmerich worked on App COVID To see that there would be different kind of like ways you could install the app, you could install it multiple times if you wanted to with a different label by subclassing it. And they would add kind of like a different label. I don't think a lot of package lead into this idea. But I think that was the idea behind the, some of the refactor there. So maybe there's a way there by using like a separate app config that have a different migration module was something like that. But the challenge is still, that will always be there is that if you start applying migrations for one path, you want to move to the other one,
Carlton Gibson 35:35
then how do you go back? Yes, migration
Simon Charette 35:39
framework is likely going to trip on something. Because it's, it's not designed in a way to when you can kind of like remove pieces that easily from from what I've seen, it
Carlton Gibson 35:51
has to sort of be off from the main trunk has to be clean, apply and clean on apply, otherwise, you're gonna go into a world of pain. Yep, quite quickly. Okay. I just wanted to get I just wanted to get your first take on that as to.
Simon Charette 36:05
Yeah, that's my take. Yeah.
Carlton Gibson 36:10
It's so good. I'm gonna ask you one more thing about heartache. No, I'm gonna let we'll have we go. I asked you about SQLite. Because there's been some changes recently that are going to come in. And I remember, after finding out that Jango Khan, something or other 2020 22, we were just talking because Simon Willison had done some benchmarks on using SQLite. And I know, Tom Dyson has spoken about using it in production as well. And I remember you sort of saying, Oh, but you know, if you need access from multiple servers, and multiple front end servers and things like this, they're gonna ask you about SQLite. And, you know, your take on that, and then the recent changes that have come in? Yeah.
Simon Charette 36:51
So I mean, that's interesting, kind of like the evolution of Israelites and this recognition just within the industry in terms of like this, like very solid tool that you can basically write to this faster did and organize it that you would, otherwise for trying to do your own fastest interaction. So I think it has the perception of Israelites in the industry has changed a lot in the past 10 years, where it was previously more of like a, I would say, like a toy database, but more something that you would use for the development or so on to a, a moment now, where folks are like, building solutions that use STL, like a distributed manner, where you can have like multiple things that are writing the same time, just use that amount. So the game changed a lot with regards to STL light, and there's a lot more interest in trying to use it. Instead scenarios that you would have possibly used more like something like Postgres or MySQL in the past. So the benchmark that Simon ran were very interesting in terms of like seeing how far we can push it. And there was, at the time, there was some discussion as well, in terms of tweaking sub mode, there was the something that relates to our transaction behave when you open a transaction in terms of like when the lock is acquired it in a recent change that was merged in was to allow to specify through a database option, how do you want it to be a basically So previously, that word was working, when you open a transaction, there was no lock that was immediately acquired, because you could have like, reasonably transaction in this game, so you don't, you don't need to add this convention. But the moment you tried to do a write, if the database was already led by something you would get to the server. And what this thing is, this new flag is allow you to kind of figure is to add the luck being immediately awkward, which can have like some impact and contention, and is not going to be working well, if you use the atomic request setting, because that obviously opens a transaction for every request. So even if you're only doing reads, which is the case for a lot of web application, you're going to run into issues. But if you want to do a lot of if you have like differences, you're going to try to push the limits in terms of delivery that might help because otherwise you might get errors that are you basically have to litter your code with try statement and link possible exception when doing writes because the database is locked. So there's something else as well as being discussed with regards to changing the right to add love or something setting. I've not followed down with that closely. But yet to set things that should make it easier to use Django with SQL lite to a larger scale. So
Carlton Gibson 39:49
since Zappia will be moving to SQL SQL Lite. I don't think so.
Simon Charette 39:58
There's not we use SQL light for something that was very interesting. We moved away from this solution. But we have a system that injects web hooks that are coming from all over the place. And obviously, you need to make sure that you never lose a web hook, right? Because if you do, then zap might not trigger and the service might not be smart enough to calculate retries for up to two or 500, or something like that. So one thing we used to do was to, we used to mount volumes directly on these boxes and pods that were ingesting user requests. And we were writing in kind of like another plug in Chardon by minutes, I believe SQL lite databases. So we would if we in the catastrophic cases where the queue was down or something like that, this thing could keep ingesting and we could offload the the machine and run a script that would open these 1000 1000 1000 database and replay. So Catholic a right to Ed's log in front of a queue written in SQL Lite. And that worked very well for us until we moved to another technology eventually.
Carlton Gibson 41:06
Okay.
Will Vincent 41:07
Speaking about API's I need to ask, Have you played around at all with Django ninja? Have you had a chance? Yep. Do you? So what last? So I, what do you make of the current API landscape in Django, right, because of Django rest framework, which is great, pretty feature complete, maybe not so active. And I hear a lot about Django ninja, where do you how do you analyze that situation,
Simon Charette 41:29
you have this exactly. But it's like a few months ago, in terms of like, what should be used new projects that are spin up at within the organization, a lot of them use Django ninja, because like, most of the time, when you spin up new projects, they're I would say they are going to be prototype, but they're like, you're going to try out things. And this the ease of development. Around Django, Ninja developers like it, they feel like they're more productive using it, then Django rest framework. For large projects, too. I, we haven't run into a point yet where we, we need to calculate, we missed the batteries inclusive of DRF, the large project that were created with Dev use it and it's even if it's sometimes it's very explicit in terms of like, like, wordplay, but like there's, there's way more like bells and whistles that with Django rest framework that you get with Django ninja. In these projects, we keep using Django rest framework, but in the new ones, a lot of them are opting for Django ninja instead, because the it works better for them. And it's, we're eager to kind of like try things that makes development easier for some of these projects.
Carlton Gibson 42:49
And sort of related question there is about the kind of the serialization part of it like that the serialize, the explicit serializers, which sort of were inspired by the forms API versus that kind of new or pedantic, inspired approach? What's your experience there?
Simon Charette 43:08
I really like you know, like, I'm more of like, the DRF person myself, because I've used it more, and I know how to find my way around it. But I think that the the appeal of Biden tick is is good. I mean, people like it. Like, it feels like the juggler was developed in the different time, right? Like with meta classes, there was no nothing of like, the nice DSL that you can get today with typing. So I get where it's, it's, it's easier, it's faster. And if we're honest here, like, like Django was less developed today with like the typing things, it would, things would be very different in terms of like, what the DSL looks for model definition. So I get why people don't want to pay the tax of, well, this software was developed with typing to exist. So I still need to calculate these things. And I mean, even with project like that Django stubs, and so as you don't get as much help from your IDE and things that you get from identic, because in the typing is easier to interpret. So yeah, that that's the the feedback I've heard from from folks that prefer to use it, use it genuinely. Good.
Will Vincent 44:21
So the last question I want to ask is about the security team in the steering Council. What do we say about these? I feel like well, let's start with the steering Council, right? Because this in many ways, replaces the benevolent dictator for life. And it's the most prestigious position to have and yet, like so what how is it? How has it been being on there? And then where would you like? Would you like to see it do more or less around making decisions like typing for Django? Yeah, I
Simon Charette 44:51
think there was maybe a misinterpretation of like, what the role was like it was loaded. discussion about like the technical board should get involved more damage Catholic dragged to technical vision of the project, and so on. While it never really happened over the years, I think it was first interpreted as more of just replacement for the fact that we didn't have kind of like more didn't have, like the court was dissolved. And we did some folks to be there to kind of break the status quo in terms of like, we need to take a technical decision here. So kind of like being us, and we'd be able to provide some guidance. So I'm seeing our Jinglu kind of like, organically evolves, and kind of like tried to adds a few things. I'm not sure like, the the evolvement from the technical board in terms of like driving division with would be Nissley beneficial. I do feel like it happens organically to a certain extent, and trying to, like the features that are going to be highlights are kind of like self imposing to certain extent. And do we really need them? In terms of like, do we need, we need like a shiny feature, every release? Is Django the point where we need to kind of like reinvent things over kind of like being stables and keeping up with the latest development. So yeah, and it's interesting, because if you have like other other projects, like for example, rails and such, the, they're very much streaming this thing and sort of like developing new things and new paradigms and so on, and Django has, has not been that. And is it okay, feels okay, to me to a certain extent to have Django catholyte Take this position? Do we fear I don't, I don't think it will make Django irrelevant by just keeping it its current pace is what
Will Vincent 46:45
I mean, I think we've seen that with, like templates to make one example, right, the fact that Django hasn't been changing so much means that now with HTML texts, it slides in nicely, right, whereas Django could have easily gone off in these different directions. And now, maybe be questioning that. So that's just one example that comes to mind about being a little slower, maybe pays off. Yeah.
Simon Charette 47:09
And there are tons of project like, for example, like just the async, dish, endocardial, you work a ton of that, with channels and such, they are these counts, like new paradigms that are being included. And we slowly get there. I'm, even if the technical board wanted to kind of like, say, like, oh, we need to guys, like go fulfill the async. Or we need to do this or that. I mean, we've got fixed kind of like release dates. It's all contributors, benevolent folks that are there. I'm like, is it going to Alpha kind of like trying to put these things it's going to shrink the community even more? In terms of trying to meet these objectives? I feel like it's the way that happens is kind of good. Like, if it's ready, we're going to make there yes, there are some features are trying to get in and focus more, because we want to put a highlight there, but it's, it's happening at more of a sustainable pace than than if you were to try to set these objects at least that's, that's my take on it.
Carlton Gibson 48:08
No, I think async is a good example. In that it's, it's, it's taken quite a few years to go from nothing to where we are, but it's actually it's quite mature now. And it's it, you know, it's not the fully async, you know, brand new web primer, if you want one of those, there are those to choose about. But if you want to add JSON async to your Django project, it's really there. And it's good, and the streaming responses and disconnect handling we've added and you know, all these things, they they're great. They're wonderful. And yeah, it's matured over time. So that's just
Will Vincent 48:39
one more I just said the security team, which you're on, I just want to shed some light on that. Because that's often held up I think rightly so is one of the gold standards that Django is fantastic with security. So if I'm a developer, and I find what I think is a bug, I email the security team. And then on your end, what does that process look like?
Simon Charette 48:58
Yeah, so we review the report, we try to kind of like assess whether or not it's it's a it's a false positive. Whether or not there's an explanation vector there. And assuming there is we're we will let the submitter know and we will work be working on our side in terms of providing a batch or with the submitter in terms of providing a patch the the problem in some cases, like if we look at like the recent ones that were we shipped like security corrective for a lot of times it's kind of like around reg X's things that are could they look benign, but like if you start to kind of like pushing kind of like crazy user input in there, they start to misbehave. But yeah, that's the process. Basically. It gets submitted. And before we release anything, we submit a request for a CTE. I Uh, to tie it to the the release itself and the commit message. And from there, all the supportive jangle versus support to diversity of Django are going to get backwards. And assuming they're affected by the vulnerability, and the author is going to be credited. But yeah, the what the security does basically is, look at these emails, make some assessment. And from there determine is this a problem or not? We sometimes we get reports for Django related projects as well. I know that the case like Django rest framework, we add some recently as well. And we try to redirect that to the folks that are involved in these projects, or try to see like, Oh, is there any other parts of Django that might be affected by a similar problem? But yeah, that's about it. There's also kind of like a recent discussions about the collusion I'll we can get more folks involved in there and ensure the parity of the of the team going forward. The I think, Carlton, you can talk more about that. But the the push there by Michael Mann three was was about possibly using the the pipeline of new contributors as a way to get more people involved in security. Yeah,
Carlton Gibson 51:17
I think my thought that I threw in there was the triage and review team might be a nice onboarding route there, because there's a kind of there's a certain trust element in being on the security team in that it's confidential. And so it can't just be, oh, I want to be on the security well, you need to be known and trusted. So the triage and review team is a nice onramp for that you're already been around the Django project for a release or two, and you're known, and there might be a nice way of doing it. But I think, again, it's important that it we think about the stability and about the continuity of the security team, as well as you know, just as much as the rest of the Django ecosystem and project. So that's there. But then what you know, one thing Simon, I have to say thank you for your service, because that's you know, it's an it's an It's a thankless task at times.
Simon Charette 52:08
Thank you both as well, like you're driving the community like that. I know. Well, people just go get very excited about just listening to Jenko chat and seeing you just revolve around the you've been just, yeah. Stars, if you're single,
Will Vincent 52:24
well, they shouldn't be excited about you know, the the security team, that's that's the thing is that the real work is happening in the dark.
Simon Charette 52:32
It's Black Ops. Work is everything everywhere. Some of it is more we're facing. But yeah, it's it's awesome to know that there's, it feels great to be kind of like in this distributed. Entity and seeing it, calculate, continue, continue to work evolve over time. Obviously, there are things would like to evolve faster, and so on, but it's, it's great to see. You said row basically, I've been Catholic, revolving around it for for more than 10 years now. And it's, yeah, it's an awesome project.
Carlton Gibson 53:08
Okay, so we're coming up on time, Simon. So I can't wait a couple of weeks ago, we had two fellas from pin planet, which is an app built with Django. And Andrew, though, is the expert. And he has had a question I promised him, I'd ask you, which is he's from an SQL background. And he's used to working in raw SQL and then when you jump to the ORM, and he's very capable and pleased to be able to construct equivalent queries in the ORM. But his point was, they often don't look very much like the underlying SQL that's generated. But you know, perhaps you have to use values in a weird way in order to get the grouping right. Is there is there any way that we can, we can make the relation between the ORM query and then the SQL more transparent is, in your view, the
Simon Charette 53:59
big challenge with DRM and something that SQL actually doesn't have? Well, it has the DRM, Query Builder doesn't have this intermediate notion of tables. So everything relates to models. Sure, they are kind of like aliases, that represents tables. But one thing that asked SQL alchemy as is that there's these two kind of like layers, like you've got the kind of like table representation you can find models to them. And you really have a query builder you have like a lot of flexibility in terms of like you can use tables directly and use like exists clause here kind of like this join here and so on. And you can then kind of like pipe that into a model representation dorm doesn't have that the, the counseling compiling part is every coupled to the notion of a model, which makes things like any form of like union or admission of like eventual Common table expression, the with statement in SQL or teams like there was discussion about a sub query wrapping and things like that. It's very, there's a fine line in terms of like providing very high level abstraction while still allowing low level injection of logic. And that's, I believe, the big challenge with DiggerLand. If you look at like, primitive that were recently introduced, like, filtered relation, for example, that tries to kind of exploit that it's, it tries to do it, and it it, for complex cases fails to a certain extent, because the ORM very much needs to be able to know what kind of joint it's dealing with, with regards to aligning them and optimizing them and making sure that the core is still make sense in the context that you make. So I think that's the kind of like biggest challenge in terms of providing more flexibility to folks that are maybe have a DBA background, we're very familiar with this drill and want to turn to Redmond to doing what it wants. So that's, that's a big, that's a problem or design issue with the current state of things. I know NC wrote up a few, like years ago that might be beneficial for DRM to kind of like be built on top of SQL alchemy, so people could like drop back to just doing these kinds of things. But that's too big of a ride for me that's out there and support multi year adventure. Right. Okay.
Carlton Gibson 56:41
So then the, I guess the question is, if do you, if you put that on the side is something that we're not going to do? Do you have a kind of magic wand that you would you know, if this if I could fix this, I'd fix it. If I could change this I change? Oh, yes,
Simon Charette 56:53
there are tons of things. I wish the we were doing a better job at Catholic buying more into removing a bit of like the extra stuff. So today most of extra is obsolete with one thing that remains is around the course of extra allows you to do a few things like select columns, but are thereby or were there but all of that can be emulated using raw SQL, the expression itself can like import from Django DB models, raw SQL and rabbit and pass it directly to filter the ribeye. And the fact this is carried around all over DRM makes it and the fact that it's like kind of like not that will support the developer cases and makes it very awkward to do so like finally pulling up the bandaid on extra Catholic deprecating most of its options and coming up with a nicer way to do reference to tables either being through joins common table expressions and such, I think good, good. Push David's even further in terms of like what is doable without going through a massive refactor by introducing an intermediate representation.
Will Vincent 58:14
Okay. Anything else that we didn't ask you about or you want to plug before we sign off?
Simon Charette 58:21
No, that was great. Just wants to say that it was it was great to be here was great as well to listen to you all. It's great to be part of the community. And if they are, again, I want to reiterate that commit to it. I want to kind of like get closer this year to either like Django, Django knots or any form of like similar projects to find a way to calculate the form of mentoring if you need you have questions about DRM, about APR, pull requests, you can you can ping me on the on the on the on track as well and do my best to try to support you through that. And I learned about DRM by just throwing in breaking things and moving things around. So it's only a matter of like, trying I guess so it's it's a lot of stuff, but it's it's certainly manageable once you you play a bit with it. Awesome.
Will Vincent 59:16
Thank you for coming on. It's truly when we started this podcast one of the guests that Simon that Carlton wanted was you you were like this mythical figure that hadn't been to like recent conferences and worked on the CRM and so to to meet you at Django con us two years ago. And then now that you're on like you were like one of the top five people Carlton was like we need to get this person on started this
Carlton Gibson 59:40
only took me 140 episodes to pluck up the courage to actually invite you.
Simon Charette 59:46
Yeah, it was I I've done that to the inviting it's glad to be here. Glad to be here and I'm regarding conference possibly I'd be able to be more present. I think that the past years but like I had to like get another surgery last year and other things. It's just And I played Ultimate Frisbee before during the exact season where out to the conference were happening. So did it work out but yeah, looking forward to possibly meet you in person.
Carlton Gibson 1:00:10
That's great.
Will Vincent 1:00:12
Well, thank you. So everyone we are at Django chat.com. And we'll see you next time. Bye bye.
Carlton Gibson 1:00:17
Bye bye