Adam is a prolific contributor to the Django community and the author of the forthcoming book, Boost Your Django DX. We discuss his work on the Django Security Team, the Django Technical Board, multiple recent Django third-party packages, and the new book.
###Support the Show
This podcast does not have any ads or sponsors. To support the show, please consider purchasing a book, signing up for Button, or reading the Django News newsletter.
Carlton Gibson 0:06
Hi, welcome to another episode of Django Chat, a fortnightly podcasts on the Django web framework. I'm Carlton Gibson joined by Will Vincent. Hello Will.
Will Vincent 0:13
Hi, Carlton.
Carlton Gibson 0:14
Hello Will! Happy New Year as well. And today we've got Adam Johnson back with us who's you know, you know who a Django technical board member, Django author. So you've all got speed up the Django tests and coming up now soon as improve your developer at Django developer experience, which we're going to talk about today and various other things. Adam, thank you for coming on the show again.
Adam Johnson 0:36
Thank you very much, and Happy New Year to you.
Carlton Gibson 0:39
Happy New Year to you. So,
Will Vincent 0:41
Carlton, you missed when Adam just won the Malcolm Tredinnick Memorial Prize? Yes, he did is well deserved. I voted for you. And I would just say the two top vote getters are the two of you. This year, so it was a very, very easy choice for the board. Thank you. That's good. Congratulations. Very well deserved.
Carlton Gibson 1:00
Thank you. Well done. Adam your contribution is massive, you're doing posts all the time, you're putting out packages, after packages after packages, you're always there on the mailing lists, you know, giving people the welcome pointers. So Super,
Adam Johnson 1:16
thank you very much. Yeah, I'm honored to receive the prize. And it came in the worst year of my life. So it's nice and Okay,
Will Vincent 1:25
we have lots to talk about, I would maybe I think there's a lot of the two of you talking but I would love to the Django technical board. Maybe we could tell folks what that is and what they do since Adam, you're on it. And that's quite a big deal. It's also a relatively new thing. And how that works with the Django code base for new releases since 4.0. came out just a couple of weeks ago.
Adam Johnson 1:46
So so the technical board is five members have voted in for each release cycle. With where the buck stops with decisions on whether things go into Django or not, like quite often, like features are clearly like a good idea. So it's okay for a few people suggested fellows to go and review burger for larger scale changes or contentious changes. A technical board vote would be held, and we'd need consensus, whether or not something would be added. Yeah.
Will Vincent 2:20
So it's if there's if it was an easy decision to not reach the board. In the same way for community that's what the Django Software Foundation handles non technical things. But the DSF has no sway over the technical port decisions by
Adam Johnson 2:33
design. Yeah, exactly.
Will Vincent 2:35
I actually I should know, is that go the long term cert releases, or is that the number release? It's every number two to three, two, or is it
Adam Johnson 2:41
so I've been on on the technikboerse. Since two, two, or two then each time since. And there's like, maybe eight or nine people who've stood over that time. So most people who are standing are getting elected at least once. It's just kind of cycling between a few people. I think
Carlton Gibson 3:04
it's kind of interesting that because I mean, one comment was made on the Django Software Foundation met DSF Members list that the elections not hasn't been particularly diverse. And it's all, you know, it's all white men, still. And part of that, I think, is because we still have a quite narrow experience contributed by the qualification for the technical books, you have to be involved in the day to day running. I mean, you don't have to have been there forever. But you do have to be active on the discussion list. And things like this. And we still don't have a very broad contributor base there. So that, you know, there's not necessarily a wider pool to draw from, which is something we kind of think about, but
Adam Johnson 3:49
it's definitely something that would be nice to correct. It's very hard because you have to have been contributing for years before you're really eligible to be on the technical boards, you have to have some deep knowledge of various parts.
Carlton Gibson 4:01
Of course, that's correct, right? Like, the chicken and egg. Yeah.
Will Vincent 4:09
Well, the DSF doesn't have that requirement. So it's, you know, you can don't have you don't even have to be an individual member to be voted on, though it is voted on by the individual members. So you do need to be known in the community. And all those things are true. I think one thing I found this past year, I think it's basically, for the first time since I've been on the board, this is my third year, we had basically all the same people reelected. And diversity is nice. I mean, I'm not planning to stay indefinitely, but it is easier logistically to have the same people because certainly for two years terms, because you kind of know how to do things. Whereas initially that first couple of months, people are ramping up. So when we my first year there was or my second year there was like half the board was new. And so a lot of time was spent kind of getting people up to speed so As internal baseball, but maybe longer term, there is something I think to multi year terms, just to kind of avoid the ramp up and ramp down. But at the same time, a recycling of voices is healthy as well. Absolutely.
Carlton Gibson 5:14
I'm not sure about this. It's not to this Jenga release every eight months. It's every is it every major release cycle now? So it's every eight months or every two and a half years? For the technical board?
Adam Johnson 5:26
I thought it was. Right, as it was to to I was voted in on and now I will in the four zero. Yeah.
Carlton Gibson 5:35
Yeah, it's kind of talking to marriage about this. Do we need a new election?
Adam Johnson 5:39
It's a ceremony that goes right past very quickly.
Carlton Gibson 5:43
Yeah, well, eight months is too quick. I think, you know?
Will Vincent 5:47
Well, I think you know, that the DSF board a year is, if it's, if I had total control, I think is too quick, too, I think there's something to, you know, perhaps, you know, three out of three, two or two to three year term, just to avoid the nuisance, because it does take up quite a bit of time. And it was something for us to discuss with our, our bodies.
Carlton Gibson 6:09
So we should give it a couple of examples of things that technical does. Not quite often, we discuss on the mailing list, and we've just come to a consensus. And it's, it's, but there are a few issues. So one that has been really important for for was a discussion about typing, because there was no consensus at all. And you know, there was a DEP and there was massive discussion on the mailing lists. And as you know, maybe in been draft PR, I don't know. But in America, and I just aren't in a position where we go, we decided to merge it. So we decided to close this. So we asked the technical board for that. And you were involved in that discussion?
Adam Johnson 6:49
Yes, that was a big one. And we came to the conclusion that Django wouldn't add type hints at the time. And that was like, a couple of years ago, at this point. And obviously, typing has moved forwards. Many of us have gained bit more experience. And so it would be nice to re approach that decision. At some point. Yeah. And yeah, I guess the second point is who's going to do the work? Because it's, it'll be a lot of work at the type center, Jamie?
Carlton Gibson 7:19
Yeah, I mean, so you've been blogging loads and loads about, you know, types? So would it be fair to say that, you know, you'd be you'd be keen now to think, again, about adding those?
Adam Johnson 7:31
Yeah, I think I probably be in favor. But we'd probably have to come to a decision that we'd only officially support my PI, or something like that, because you will need a plugin for type checker to tell it when you make a Django model, the field definitions map onto these types.
Carlton Gibson 7:50
Okay. I mean, like, one thing that I've kind of been floating around is around the widget API, because you know, there's lots of, there's lots of areas there, where it's like, what do I get? What exactly do I get out of value for day to day care? And, you know, I could really, I could really appreciate some typing there. But my kind of thought is, Oh, I wonder if we could do this just in save the widgets file, you know, just in one kind of module, where it's, it's the kind of leaf you know, that the widgets don't really impinge on anything else. They're not impinged by anything else. Could we type those? And that might be a sort of progressive way of doing it?
Adam Johnson 8:25
I don't know. Yeah, progressive would be more will be how I like to see it happen anyway. Yeah, all these leaf modules, like all the things in Django utils. That'd be perfect candidates to begin with. But we'd need to discuss, is there value in adding just a small amount of tight pants initially? Or is that going to make it harder for people? Because then they're going to have, oh, we're trying to use the tight pants, but everything else is showing up as any and causing us headaches? Some research is definitely required.
Carlton Gibson 8:56
Yeah. Okay. Interesting. Interesting.
Will Vincent 8:59
Something else technical, I'd love to mention. So as I went to bed last night, I saw there was a, there's a new security release. And this morning, I see in the notes, Adam, you've done a whole blog post on it, which is insane. So I wonder if we could briefly talk about that. And that's a chance to mention that you're also on the security team of Django, which is a separate group for a few months now. And
Adam Johnson 9:21
oh, only a few months. I think Colton invited me in September.
Will Vincent 9:27
Oh, wow. Well, I just assumed you were part of it. Last couple of years.
Carlton Gibson 9:31
Let me give a bit of background. Let me just give a bit of background now. And so there's, I know about 10 members, some of the more the older contributors that have been around for you know, super knowledgeable, but not necessarily involved in Django day to day anymore. And you know, there's a few people who are still very active like Mark this Holtzman and flowing upon. But we're finding that you know, it was me and Maris and maybe Marcus maybe Florian Comment and we just needed some some more people directly involved kind of regularly. And so amongst the security team, we talked about it with Adam and Nick Pope came up as and so we invited both of them and they come in. And they've been super helpful and active because we get a lot, not a lot of repair Money App reports, would you say we get admin at
Adam Johnson 10:21
say the average is one every two weeks? That's really actionable. Would you say that's about it?
Carlton Gibson 10:27
Yeah, I mean, it's not it's not a lot, but they all need dealing with they all need looking at and it's not easy. It's really not easy. You know, I mean, I'm not a security expert, and a thing comes in and it's I don't know, is this. And without the security team, I there's no way I could handle those, you know, those issues? Like yeah, yeah. Okay. Some of them. It's easy. Not that's not a problem. Some of them? Oh, yeah, that clearly is, but then this vast area of that's a really difficult question.
Adam Johnson 10:57
I think I really liked seeing how everyone like filled in the holes of knowledge and ideas and solving these issues. And, yeah, because we'd have a lot of back and forth before even a pull request is open and be like, Oh, we could do this. We could do that. Yeah. So there were there were three fixes into this release, right? Yeah, shall we? Shall we run through them quickly? Okay, so the first one was a denial of service possibility in use the attribute similarity validator. Now that's a mouthful, isn't it? That's a password validator that checks that the password given isn't too similar to a user's other attributes, like their email address. So you they're not just recycling the email address, or adding a one and exclamation mark at the end for that password.
Will Vincent 11:51
Yeah, who does that?
Adam Johnson 11:54
I think quite a lot of people. That's exactly what I do. Yeah, so it's active by default when you do start projects. And but who was it that reported that one, Chris Bailey reported that if you submit a very large password, then it has kind of n squared or exponential runtime, I'm not sure which you have to submit like a very large password like 100,000 characters, but then that can lead to like many seconds of processing on a on an average server. And you know, submitting 100,000 letter A's over Jesup is not much of a request body for the client to send. But then it can result in in many seconds of processing. So it becomes like a denial of service factor. Someone submitting loads of these requests could cause your servers just hang up. So the fix that I came up with, along with Florian and Colton and Marius is to avoid checking the similarity. If password if the password is like so much longer than the attribute value, that it's not going to be some regarded as similar anyway.
Will Vincent 13:03
So how do you come to that conclusion? The right I mean, are you like, Who do you didn't? Did you come up with that? Or is that something that other security researchers have used as a best practice?
Adam Johnson 13:12
And we had to come up with this one, I don't think there's much groundwork for using this kind of algorithm.
Carlton Gibson 13:20
I mean, I mean, this was quite an interesting example, because Adam had an idea for to this well, you know, we'll take it like if it's, if it's a plural, you know, a multiple, sufficient multiple cut it off. And then there was various back and forth and different approaches, it was really nice. This Adam's point about, like, everyone throwing in their ideas is that there are two or three different takes. And it's like, yeah, actually, this one's quite sweet. And he has a nice ratio that this is a realistic cut off at different scales of input and yeah, and whatnot.
Will Vincent 13:51
Cool. And then there's two more.
Adam Johnson 13:53
Okay, we'll go through them a little quicker, because I'm a bit less familiar. And the second one was with the dict, sort template folder, this is one that I've not seen used, but it takes a list of dictionaries and then sorts them by a key or perhaps an attribute and, and security researcher at Sona source, Dennis, print growth submitted as a report back, you can use it to access like arbitrary attributes. And inside of a string, you can access the characters of the string one by one. So if a user manager submit, which order to sort your list of dictionaries by they could sort them by character zero, then character one then character to see the sort order changing. And then they could infer the value of that attribute. And if that value was like security sensitive, like a password or API key, they can kind of infer, okay, this, this user appeared at the top of the list, so their password starts with A and then when I sort by character to that in the middle of the list, so that's probably something around L. And then it can kind of combine that with other info to get a statistical guess at what their, what their secret value is.
Will Vincent 15:00
Yeah, like you phrased it in your blog posts a good example of Don't Repeat Yourself causing problems like that. Yeah.
Adam Johnson 15:06
Because the problem came from the filter, reuse the Django variable class, probably over time variable gain more abilities. And then great, that has has led to the ability to index characters in a stream and do the sorting. Whereas it was never advertised for doing that, and probably no need to.
Will Vincent 15:25
Well, I think the more common example of this right is with URLs, where why you would use a you UU ID. So instead of going user slash one or right, because as you said, it could apply to bank records or something important, you don't want someone to infer the order, or the number. And then number three, I can read it. So there's a potential directory traversal via storage. Dot save class, which sets a file store JP, I think, yeah, so when you're uploading a file? Well, I think it's just it's interesting to talk about this, because, you know, I think like many I, I see the reports that come out, and I'm just like, Yeah, it sounds good to me. And unless it's something I directly have, don't get a chance to fully dive into it, or even know the process behind it. So yeah,
Carlton Gibson 16:05
these these file driver, these file, directory traversal ones has been several of those over the throughout the, you know, the upload handlers, and then you know, blah, blah, blah, and there was a number and this is kind of like the last one, and you'd have to use the storage directly, which kind of nobody does. So it's already been bullet proofed around the outside. But the idea would be that, you know, you go from the media file, the media directory, where you're meant to be saving it to somewhere else on the file system, which, you know, you're not supposed to have access. Now, you know, if you do everything right, that can't, you know, that's already ruled out by various other bits in Django, but, you know, this is the sort of last bolt down on that area, there might be a bit of a rewrite coming in future, who knows,
Adam Johnson 16:48
saying it's the last possible directory traversal? Bug is very brief.
Carlton Gibson 16:52
We'll know in that little bit. It's the last possible one in that that cluster of you know, that, like cluster of upload handlers and files and week, because we've gone over them, no doubt they'll be
Adam Johnson 17:03
next week. I think there's been like sufficient eyeballs on it now. It's pretty, pretty good. This was again by Dennis Brinkerhoff, and Sona source. So I guess they were running some kind of scanner or just deciding to audit open source project. So they found it. That was cool.
Will Vincent 17:21
For something else, just to shed light on all the things to do Adam just PRs to Django itself. So 4.0 came out, I guess, Carlton, this was in your your weak notes thing, right saw where I don't think it was on the release notes where you listed all the active contributors. And Adam, you were one of the largest ones. We'll link to that. It's not in the release notes, right? I don't think the formal No, no, no, no, no, no, no.
Carlton Gibson 17:43
So I've so. So one. So what I've got for 4.0 is a breakdown of new contributors of people who like I've called them on a roll people contributed in the last in 3.2, as well, and then people who didn't contribute in 3.2, but had previously could put those in the category of Welcome back. And what I want to do is write that up, and then put that somewhere I don't on my site or on Django project.com, if we can find a suitable place for it, and then keep that rolling for 4.1. And as we go forward, that would have happened already, if it weren't for, you know, 2020, or the huge excuses. But that that's coming. And the idea there is just to have a bit of fun calling out contributors and you know, particularly for new contributors, you make a contribution, if can we find a way of just flagging up and, you know, it was, it was on my list to get done for the end of the year. And that just didn't happen. So okay, over the next month or so I'll pull it up and finished, finish that off and have that go going forward. But that that the idea there is to do something to call out contributors a little more than we have, you know. I don't know, it's just a bit of fun.
Will Vincent 18:51
It's I think it's important, though, I think it's important to highlight those things. You know,
Carlton Gibson 18:55
I think it's important because of exactly this problem of diversity, we've already talked about just just now, like, it's alright, if you're sort of economically privileged, and blah, blah, like you can contribute loads of time to open source and you can spend the time to get contributor record and you know, but if a lot of people, they need some recognition for it, you know, so I've made one contribution. did that, did that pay back to me? Well, no, it didn't, because it just disappeared into the void. Whereas if we can put it on a website somewhere, and then they can point to it, it adds a little of, you know, for for someone getting into tech, it adds a, a validation point that might help them get a job and then, you know, I can contribute, you know, I can continue, we need to, to, to do things to smooth the pathway there, or we're never going to change the situation we're in it's always going to be slow, but we're not going to instantly have a super diverse contributor base. But if we don't do these things, we're never going to have that contributor base.
Will Vincent 19:47
Well part of it, Adam, I mean, cuz even just your, your commits to Django, there's a wide range of how, how big they are right? Like I think you tweeted there was one template, like it's like a one line thing with you and Maurice, you know, And then you have, you know, signal receiver function stuff. You know. So there's, I think it's great that you highlight that, because that's something where it seems small. But like, that's exactly onboarding way versus going into like a core piece of Django. And thinking that's what it takes to do a PR.
Adam Johnson 20:15
Right? Yeah, this this one line change, I just was reading the books, I found that there's a class that exists, but the setting that you'd put that class in, didn't document it, it said, Here's the list of built in and listed to the three. So it's just adding the third one on that list. And I think there are like a number of these small things out there, you might spot when reading the docs, and it's great to just go make that change. So it helps everyone else in the future.
Will Vincent 20:41
Yeah, I'd also I wanted to ask you, I mean, I know you're a consultant. And so you poke around a lot of code bases, like how do you find so many things? I mean, because I guess I'm just not using Django, to the full bore the way you are. But I mean, that the range of PRs, you have SQL light signals, templates. I mean, does it? Is it client work? Is it a Doc's mismatch? Or is there any pattern there? Because you find so many?
Adam Johnson 21:05
Yeah, I guess they just end up going through a bunch of different stuff trying to solve people's problems they bring to me. And I also enjoy reading the code. And I sometimes take that as first reference, even skipping the docs, where I'm like, Oh, what's that method on the storage? For example, I have Django open in Sublime Text window all the time. Yeah, I guess the third thing is I'm always just a find finding it fun and expanding my knowledge and my curiosity by diving into things, and not really, not really looked at before. So like SQLite, I was just interested in SQLite a bit more, because they came out with that new release that SQLite has strict tables now. I was like, Oh, I wonder how how good Django SQL lite support really is? And could you could you get that strict thing in there. And when I was reading the database back, and I came up with that optimization. I was like, this seems fun. I could do it. I love that. And Nick Pope was was very good at reviewing and getting helping me out.
Carlton Gibson 22:08
So I, while we've got you while we're talking about the program and contributing, so you're, you're one of my sort of go to Rm knowledgeable people, you know, that I can flag up on a PID? Hey, Adam, can you look at this? You know, obviously, you did all the work on Django, MySQL, and you've been contributing there for a long time. One of the questions we always have from prospective new contributors is how do I get going contributing to the rim? So can I put that question to what might you what would be your kind of initial thoughts? If somebody is coming? You know, somebody wants to get involved and they want to get involved in the ORM? Specifically?
Adam Johnson 22:41
That's, that's a hard one isn't that I think the problem with DRM is that like, it does, like 99.9% of all the queries people need. But most developers probably only know how to write like 99% of those. So there's this kind of ton of extra features that are hard to discover. And some people might be tempted to just try and help but it's already there in some way. Yeah, I don't know if there's an easy path to the RM. I know, there's like a James Bennett three hour talk at a Django con. That is like a deep dive into the ORM. And it's a few years ago, but the fundamentals won't have changed. If you can sit through that you're definitely going to be a good arm contributor.
Will Vincent 23:27
I've tried three times. I still really want to it's it's no slag on him. It's just It gets deep, fast. Yeah.
Adam Johnson 23:37
And it's a bit of a trial by fire.
Will Vincent 23:39
Maybe that's a New Year's resolution for me.
Carlton Gibson 23:42
Okay, cut what cut one slightly different angle, then you said a lot. There are lots of features that are kind of hard to discover. So like, you know, do you think, do you think there's, there's a bit of a doc shortage in terms of learning more advanced SQL features or SQL features, and how to use them with the ORM, like annotations and aggregations, and then the expressions API, and, you know, all those, you know, output field types, and blah, blah, blah, all that all that stuff, which is, it's documented, but there's no real easy learning path for that, I would think,
Adam Johnson 24:19
yeah, definitely. And one, one thing I I've liked and tried to promote a bit is check constraints. I've done a number of posts on that. If you read the check constraint docs in Django, like purely a reference, they're not going to explain to you what a check constraint really is, and how it's defined on the database. And we don't really have like a topic guide there. So if someone was looking for a Topic Guide that be to contribute, that'd be a good place to start. And perhaps that's a good first RM contribute. It's like in migrations, we have the reference for all the different operations and the Topic Guide of like how to do X in migrations. is actually where you can learn how to really use that stuff like that, that has a few topics now but could definitely be expanded. So how would you add an index concurrently? Like we have that now for Postgres in Django, Contra, Postgres, but that's definitely not a Topic Guide. Like, here's what you should do to use it. Okay, so there's all these features.
Will Vincent 25:23
Yeah, Carlton seems I have a lot of questions. I don't want to overrun you. Do you know? No. Okay. I wanted to ask you this. So. So your new book, was it boosts your Django developer experience is that that's the title.
Adam Johnson 25:34
Yep. You've reached your Django the developer experience. So
Will Vincent 25:38
you've, I've seen you I don't know if it's on Twitter, your blog post mentioned like book driven development? Because so could you talk a bit about that? Because you've put out I mean, we have a long list of notes, so many projects, posts, but as you've been writing the book, so I'm really interested in that. And if you could expand on that
Adam Johnson 25:55
topic? Yeah, absolutely. So the the book is just trying to describe and like categorize all the tools that aren't Django itself, but like live around it that can really help you, like accelerate your development. And I guess book driven development is where I start writing a section. I'm like, Okay, I think I know, the best way that I've seen to do this, and I get halfway, I'm like, actually, you know that. I'm just describing some snippets that you could copy, paste. Instead, it would be better if there was a package out there that would would would combine all this for you. So like a good example, there is a Django rich package I just created to integrate Wilma, Google's rich that does nice terminal output with Django. And currently, it only provides like a new management command. Yes.
Carlton Gibson 26:44
Is there a is there a test runner coming for that, because a while ago, there was a PR to get colored output into the Django test runner, and it didn't make it because of maintainability concerns, but perhaps a third party package wrapping rich as a test runner, that'd be a nice thing to see,
Adam Johnson 27:01
it would be a nice thing to see, I did briefly look into it, and was like, maybe the package could do this. But a unit test is really the blocker there. Because you've got to override this class and that class goes next test results. And it's debug results inherit from the text results. There's like a chain of like four classes that refer to each other and subclass. And it's, it would be a pain. Okay, interesting. I think maybe an approach would be if there was a unit tests extension package that use rich, then Django could inherit of that. Okay, if it was then the rich package only provides a management command that gives you the way to wrap Django is output with rich and then turn it off if you're piping into a terminal. Another terminal Come on,
Will Vincent 27:46
in what about this the Django browser reload? I mean, that was one that I know it came out. And I guess, asking the two of you, you know, is that a future? Core Django thing? Or what are their just clear limitations on that, because that's pretty helpful.
Adam Johnson 27:59
Sure, um, briefly, Django browser reload is a tool that you install. And then in development, whenever you hit save on a template, or save on a python file that causes the server to restart, or a static asset changes, then it will hit reload on the browser for you automatically. And it does this with a different approach to what I've seen before, in that it sets up a very small, like JavaScript based listener in the browser that's listening for a stream of events from a Django view that captures them. And then that that JavaScript worker, it hits reload on on the most recently open tab that you have field development server. So if you've opened five of them, it's only going to reload one. I was inspired to do this, because I was running about dx, I knew that this is a big problem. Like people have this in JavaScript, they have like even the hot hot replacement where the page doesn't reload, it just swaps in one section. But there's not really been something I've seen in Django that works amazingly, I came across one package, I think for fast API. And this was using Selenium to control it. And I say that there's got to be a simpler way. I don't want to force everyone to install selenium, and use a Selenium development browser. To do this. There's got to be a way in JavaScript. And it didn't take long to fumble through the JavaScript API's and come up with something very basic.
Will Vincent 29:21
That's great. Now, that's really interesting that that is a common problem. And that's an elegant approach, which as time will tell, but I don't know why it wouldn't work.
Carlton Gibson 29:32
It would be cool to have an ankle that that's the interesting question is what makes it in what doesn't? And you know, I mean,
Adam Johnson 29:39
yeah, I would like to know, in what situations it doesn't work before we really go ahead and consider
Carlton Gibson 29:46
it's the sort of thing that's suitably useful that there's a case to be made, right,
Will Vincent 29:50
I suppose. I mean, there's almost I don't know if it's a conflict of interest, but if it's your own package, and you're on the technical board, and the security team, it's not really on you To propose that it'd be integrated, but I guess when it's a third party package, you know, people can use it on their own. And if there are issues will come to the fore.
Carlton Gibson 30:09
So many times we have like, the standard argument, the standard line, the standard approach is, can we get this feature into Django? Well, can you put it into a third party package? And can we see how that goes for? Quite a long time. I mean, you know, Django rest framework is 15 years, 10 years out, you know, it's, it can live in a third party package forever. And so what's the case for bringing it in, you know, that's Django debug toolbar, that would be in a candidate, but that's never going to get merged into core. You know, these kind of hyper useful pack third party packages, they will, they can just stay as third party packages. I don't know. It's but we can't just go write new feature bam,
Will Vincent 30:54
can we can ask some more about about the book, I mean, sort of you've I think you've put Well, I've had a chance to look at it as Carlton. But you know, topics covered because this is your second book, you also have your speed of your Django tests. And this will come the book will be released a couple days after this comes out. So we'll have a link, everyone should pre can they still preorder it up until the release?
Adam Johnson 31:14
You can pre order it right up until the hour it's released, it'll be released at 12 noon, GMT, that's the same as UTC right now. And on the 10th of January,
Will Vincent 31:25
this will come out we're recording on the fourth, it'll come out in the fifth. So we'll have a big prominent link to that. But anyways, yeah, to the book. So what was, you know, cuz I find the question is always what not to include, right? Because the first draft is like, let me just dump everything in. And then it's like, what to strip out? And then it's like, okay, and is this actually what I want to say, as you were, you know, with these areas, you start to give it a beginner's luck with some experience. And then sometimes you come up with different approaches.
Adam Johnson 31:50
Yeah, so I tried writing one chapter for each kind of like, broad area of the development of a Django application. So it starts off with like, looking at docks, and then how you make virtual environments and manage dependencies. And it goes on to other things like later, it's things that are actually in your code, like your settings, and models and migrations. And so yeah, it's about everything that's like, alongside the actual writing of Django code, there's not much like here's how to build an X in Django. But it's here's how to make it faster few to build x in Django. So faster, but more bug free. You're very right about cutting things, there's so many things that are actually involved in the process. One thing that's helped me a lot is have it edited, dropped file in the repo with all the topics that I've dropped in bullet point. There could have been like a whole chapter on get a whole chapter on GitHub. Yeah, a whole chapter on tech steadily. I've
Will Vincent 32:50
always Ghost Ghost chapters to have to ignore it. Why it's yeah, it's a combination of, yeah, it's like, oh, this is hard or like the deadline? And you're like, yeah, if it's important. I mean, I have a whole whole huge thing on logging for my professionals book. And I'd still like to put it in there. But it's also like, I just, I just got to ship it.
Adam Johnson 33:10
Logging could be its own Well, right. Similarly, like I, in this book, I say, Hey, you could go and steal my pie. And then it's like, I can't cover type pins here. There's no real way of even starting, so is just some links to resources.
Carlton Gibson 33:24
I mean, it's, there's a question because you did this whole series of posts on on typing, I did wonder if that was kind of, you know, drop blogging, the first draft of there was going to be a typing book come out of that. But
Adam Johnson 33:36
I have, I've considered the I wrote those for my own education, as well as to like, help push typing a bit. Because it should be possible, I think, to like, search, how do I do type hints for this Python feature and find a reasonable explanation. So I just tried to cover many of these spaces. And I think a book would just be like a bundle of these posts that you could go read like a cookie type and type in cookbook. Maybe that's
Will Vincent 34:06
why I love that you're doing these not just advanced, but related to Django, but not Django itself. Up approach. I feel like that's maybe approach I should have done. Because I struggle more with the profession, my more advanced books than the beginner one, because the beginner when I feel a little more confident saying, I, you know, I don't need to dive into everything, like just sort of, trust me on this, or, you know, here's how you can go deeper. But the more advanced ones, it's still kind of the same thing, but it's harder to do. I feel a little worse about waving my hands, or maybe I'm just not as clear on it. And I'd have to say, you know, okay, here's three ways to do it. I like this one, but I don't feel the same level of confidence on that as I would on the beginner thing where it's like, we'll just gonna backfill a little bit here. So I'm incredibly impressed that you're I mean, of course, you have the background to write these books, but I feel in some ways they're harder to write but But taking a more from the side, you know, not just doing like the most badass Django project ever approach like that's, that's a recipe for failure. That's like building a Jenga puzzle, which is what I have in my professionals book don't do that.
Adam Johnson 35:13
Well, I tried to fill in the gaps of what, what materials out there. It's very easy to say to someone, hey, go set up pre commit. And then like the pre commit Doc's don't say, here's how to do pre commit on a Python project with flaky ISO and black. And where's that? That's, that's what most people want. That's what Django does. And there could be some opinion. And I do respect your books will. And I think the fact that you're a bit uncertain about what's going into the professionals book is just an indication of that being senior right. And senior you have to make trade offs.
Will Vincent 35:49
Yeah, yeah, it all it all depends. That'll be my, my fourth book, right, Carlton will just listen to all the unknowables. I
Carlton Gibson 36:00
think your point about pre commit is a good one like because you go to the pre commit docks, and it takes it can take all day to get it set up properly. Whereas you can give the recipe look here, bam, this is how you get pre pre commit going into Django problem your Django project quickly. So one thing I wanted to ask you about, she made it seem to to manage so many open source projects. And part of that, surely, it must be tell me is that you've got this tooling kind of lined up to automate the repetitive parts of managing a package is
Adam Johnson 36:33
absolutely, I use a tool called My repos, which I guess is a bit is a bit weird, because it's written in Perl. And it's not always so easy to use. But it lets you like run many run commands in multiple directories. And it's a Git aware, so it has like a status command if you need that. So I kind of treat all my open source packages as one large repo with like, some variability in each of them if necessary, but as little as possible. So if there's a new Python version, what I'll do is actually write a script that will make the appropriate changes in the right files using SD that's a rust based replacement for Sed. And so I just run my repos and run that script and do that in all those files, so as long as everything stays relatively uniform, I can do, I can upgrade, you know, 33 packages, I think it is now in the same time that many people could do one or two.
Will Vincent 37:39
So it's like boost your GitHub dx. I guess? That's a That's a great question. I was wondering the same thing. I'm, I'm trying to like nuke Open Source Repositories I have because I just can't deal with maintaining them and, and so I love seeing that you're going the other way. I mean, your output, honestly, it just makes me tired. But I think I'm just tired the kids but just like, oh my god, like every you know, cuz your weekly posts, which people just sign up for, it's like up a couple packages, couple blog posts, a couple core commits, you know, plus everything else I do. It's awesome. It's It's honestly, again, it gives me energy, though, to see that you can be so
Carlton Gibson 38:20
I just wanted to ask you about one more package, which is Django upgrade. Can you tell us a little bit about access? That's
Will Vincent 38:26
Oh, yeah, that's very exciting. Yeah.
Adam Johnson 38:28
Okay. Yeah, that's worth mentioning. So I wrote this between August and September. It's a code rewriter that will upgrade your Django syntax, and it knows about some changes as far back as 110. Because they were there. Bruno, Allah wrote a version of this before called Django code Mont. And this used Instagram's lib CST, which is their concrete syntax tree library. And there are a couple of problems with that underlying library, though. First, it's like incredibly slow, because it's a Python parser written in Python. And it also like turns features on and off depending upon Python version. So it's like, go to these branches. And, and then the second would be that they don't really update it for new Python versions, Instagram is still running their own fork of three, eight, as I understand. So there's three, nine and 310 syntax that's just not supported. So the end result is that Django code model is like, okay for most people, but it's not something you can add in the way that I like it, which is, you know, it runs through on your pre commit hook. So you never can commit old syntax. It will just get rewritten when you try. And so Django upgraders is a rewrite of Django code, but it's using the same approach that talk pi upgrade uses pi upgrade just uses the built in Python syntax tree and tokenizer to figure out what changes to make Django upgrade does the same thing. So if you're running it on Python 310, it automatically supports all the features in 310, because the Python standard library will support
Carlton Gibson 40:14
a little play with it as T Stafford's gets quite sort of in depth takes, it's quite slow going. I'd say.
Adam Johnson 40:25
It was very entertaining. And especially the especially the going back from there as T two tokens, because when you're the token level, you're basically manipulating like this thing. All we know about it is it looks like a name. We don't know what it relates to in this in the actual source tree, is it a variable name or a class, name and assignment? So you're fiddling around with those tokens? It's quite fun.
Carlton Gibson 40:54
Okay, so I have I have great hopes for Django upgrade in that one of the big problems. One of the great strengths of Django is the Stability Policy for the the API Stability Policy, then we don't break stuff unless, you know, there's a good reason to do so. And I was asked on Twitter today after it, I was like, you know, because I was releasing 2.2 as well as 3.2 and 4.0. As he stood on 2.2. Like, wait, you know, because I don't use the LTS myself. And I tweet out and it's a, you know, agency, a couple of big agencies adapts, Jamie replied, and then Tobias from Cactus replied, saying, You know what, we use it because our clients, we it's very hard to justify the eight monthly upgrade, was it to yearly upgrade, we can, we can do that. And other other applies would like look at stability, we use the LTS for stability. And one of the big challenges is all we want to deprecate this, but people won't be able to upgrade. And, you know, which, you know, when we changed, I don't know, to using V had to import the view, rather than just use the string view name. I think, you know, actually, we lost Instagram over that change, you know, they weren't prepared to go through. And so they fought Django and you know, they're forever on a fork Django now rod not on mainstream Django and maybe they'd have gone anyway. But could we avoid that? And well, one, one hope of avoiding that kind of breaking changes, automatic scripts. So when we do introduce a deprecation, there's also a matching script that will go through your codebase and correctly update it. Is that is that the pipe dream too far? Or Is that Is that realistic hope that we might have?
Adam Johnson 42:36
I definitely share this hope is is one thing to like deprecated and add a warning. And then that warning just shows up in in your test suite. It's like here's 1000 things to go change. That's no fun. So yeah, there's definitely a place for this. And I'd be excited to see it going forwards. I couldn't spot anything in four zero, that can actually be automatically rewritten. Without huge amounts of effort. So definitely a balancing act. It's okay to change as info zero that we can't meet, we then
Carlton Gibson 43:14
say, here's a tester for you for for one, like we brought in the zone info timezone changes for Chango 4.0. And there's this Django dot you utils dot timezone dot UTC constant, which is just, you know, we should get rid of that, ideally. But that's everywhere. And so is it worth the chat, we could just leave it there forever, but we are going to get rid of it, we need to get rid of it for you know, before 5.0, because it needs to go at the same time as HP it said. So, you know, could we be? Can we use PI upgrade to to make room a deprecate that remove it, but be make removing it, you know, just run the script, that would be a cool thing.
Adam Johnson 43:57
That That one's definitely easy to change from ASCs perspective, because you can see it being imported. And you can be like, Okay, what do we replace it with? And I guess we just replaced it with the the one in Python state time module, right?
Carlton Gibson 44:09
Yeah, so exactly that it's just replacing with daytime daytime, UTC, but like it would be, it'd be nice, because my only reticence about removing it is that it's everywhere. You know, everywhere in the ecosystem. People have been using that. So if there was a script that came along with with Django 5.0 That said, Hey, you got to remove this, but just run this and it's done. That would be an amazing proof of concept of of this is the future of upgrading Django, because stability right, that's, that's the USP.
Adam Johnson 44:43
Yep, definitely. And there's, there's a good argument to remove it right? Because it looks like it's a different thing. And once it's just an alias for pythons one, all you've got is like the potential for confusion.
Carlton Gibson 44:54
Yeah, so why are we keeping it We're keeping it to avoid the churn but
Adam Johnson 44:59
that's than that that confusion can stay there forever and tutorials and people's code base snippets they get copy pasted. Why should be using Django is 109 is exactly the same. It's just an alias. Yeah, exactly. Okay. One. One thing I found in Django upgrade is some historical changes where there were aliases for Python built in functions that have been rewritten. And they weren't ever documented. But when they were removed, I found the old commits and people were debating, should we really be doing this? And we're definitely better off that we did now. So
Carlton Gibson 45:36
Okay, interesting. So we should push that for 5.0.
Adam Johnson 45:39
It's been decided here on
Will Vincent 45:46
the books coming out, at least for me, there's always it's like, it's not like a downer after something comes out. But there's sort of like a loss of purpose. At the same time. There's, there's like, Oh, thank God, like some I can worry about something else. So long term. What else? Do you know what? To work on projects, books like pie in the sky? Things you have planned around Django?
Adam Johnson 46:08
Well, I've taken a break from client works since July last year. So I'll be getting back into that. And no doubt diving through people's code bases or spots and new things that I'm interested in changing. Yeah. For the book. My next reading project will be to go update the speed up your Django test one for 4.0. Yeah,
Will Vincent 46:28
I got you on four. Oh, you got me on three, two, but I got you in for. For one of my books. I'm still waiting. But yeah. Well, I double checked, I was like, I beat Adam and something.
Adam Johnson 46:47
Congrats, it's not
Will Vincent 46:48
it's not a big change. It's not a big change. It's a pretty small change. Yeah, at least. I, I think we I think for years as well, right? It's I don't think I remember,
Adam Johnson 46:57
I, I'm lucky that my books aren't like, Go run exactly these things and will definitely work for you. They're more like, Hey, this is something that could work.
Will Vincent 47:05
Oh, my God, it's, you know, it's like, it's a test of something because I get emails and like, there's an error. And I'm like, Are you using the right code? Or did you pin repositories? Like, you know, I have 99.9% Sure, you're making a mistake. But I just like, sometimes I have to just like, wait a couple of days and be like, sure you wanna check? You know? So it's good and bad. It's like, I'm pretty sure I'm right on this. Because I've put it in this little box. But anyways, yeah, yeah, you don't have those. I envy that. Like, yeah, like, could do this. Like, it might work for you. When I go through my updates, yeah. Like, I literally, I've thought of running a script to like, check everything. But I also, for now manually go through everything as much for like the text part. Because I, you know, he's kind of when I when I read old things I've written and then like, I could rephrase the header, improve it. So it's like a forced update mechanism. I don't know if that's what you do. If you go through and read the text, or you have you probably have a test suite, you can run and just see what's broken.
Adam Johnson 48:07
You think too much for me? I like definitely go through and read things in speed up your Django tests, at least because some features were still underdeveloped. I signposted. Hey, no Django 4.0 You'll be able to do this. So remember updating it and checking whatever there is still applicable?
Will Vincent 48:27
Yeah, I do that too. Yeah, I tried to say, I think this is coming. But I mean, also, I think, you know, it's good to have some distance from it, right? I mean, eight months is hard. Which this the first last year is the first time missed an update cycle. But it's almost too close because it sort of needs some distance and yeah, and things change to like, revisit the same stuff. Anyways. Yeah, that makes sense. I wish you well on the update challenge. It's like a band playing the Greatest Hits is how I look at it. You know, it's like some for me like new stuff and then some for everyone else, which is updating things. I've already written greatest
Adam Johnson 48:59
sound like that analogy.
Will Vincent 49:03
Yeah, it's like, you know, if nobody cared, I wouldn't do it. So anyways, not to make this about writing books, Carlton, I know you love those.
Carlton Gibson 49:10
It's just I like them because they remind me not to do it.
Will Vincent 49:17
Anyways, we should, we should wrap. Okay,
Carlton Gibson 49:19
let's wrap up the items. Thank you so much for coming on. Good luck with the book launch everybody. Remember, you can pre order until the tent. Boucher
Adam Johnson 49:28
Django, dx 10% discount.
Carlton Gibson 49:32
Thanks for coming on. And thanks for everything you do to the Django for the Django community and well done again on credit Ward couldn't think of someone who deserves it.
Adam Johnson 49:40
Thank you very much for having me.
Will Vincent 49:42
All right, everyone. We will are back on our every two weeks schedule with new episodes on Django chat.com and chat jingle on Twitter, and we'll see you all next time. Bye bye bye.