Keryn (Kez) Knight is a longtime contributor and current member of the Django Triage and Review Team. We discuss the rough edges of Django, async, error pages, optimization, and the lifecycle of Python and Django.
This podcast does not have any ads or sponsors. To support the show, please consider purchasing a book, signing up for Button, or reading the Django News newsletter.
Carlton Gibson 0:06
Hi, welcome to another episode of Django Chat, a fortnightly podcast on the Django web framework. I'm Carlton Gibson joined by Will Vincent. Hello Will.
Will Vincent 0:12
Hi, Carlton
Carlton Gibson 0:13
Hi Will. And today we've got Kez Knight with us, who's a longtime contributor Django, member of the Django to charge and review team and, you know, general member of the community. Hello, case. Thanks for coming on.
Keryn Knight 0:25
Hello, thank you very much for having me. Appreciate it.
Carlton Gibson 0:28
No, it's a real honor wanted to have you on for a while. Let's, let's start, we always kick off with the backstory. So tell us about about you and how you got into programming and how you found Django. Because
Keryn Knight 0:39
I got into programming accidentally, by when I was a teenager, we kind of was looking at doing various things on the web. And we kind of learn a bit about it during school. And so I learned HTML, and then then nascent CSS stuff that was happening, and accidentally formed a career around that I managed to get my first job doing that kind of thing that led to more kind of full stack work where, you know, be involved in the back end. And a lot of the front end back when the front end was much simpler than it is today, I must stress that and started out doing like PHP back end stuff. And ultimately in around 2010 sat down to try and make something more maintainable when we were working in a web development agency. Obviously, there's a lot of juggling of projects there. We were looking for something that kind of fit the bill and was easier to onboard and revisit later. And we revisited a whole bunch of different frameworks, cross languages, and ultimately settled on Django as having the least cons of all of them. And I've been using it ever since.
Carlton Gibson 1:58
Okay, and so in an agency environment, is that the primary way? Yes,
Keryn Knight 2:02
yeah, we were an agency for a number of years. And then we pivoted towards doing a specific product later in time.
Carlton Gibson 2:12
Okay. And the key bit for the agency stuff is you've got lots of projects on the go. So you're always cutting and cutting back and coming. But they got to be the same in some sense, because you can't reload relearn something bespoke for every single project.
Keryn Knight 2:25
Yeah, I mean, a lot of our work was very bespoke. You know, obviously, everyone's supplements these things with your relatively standard CMS or CRM kind of work. Because that's reasonably quick to turn around if it wasn't too complex. But you had things like WordPress, kind of taking control of those kinds of markets, certainly for the CMS stuff anyway. So that meant that we moved more into kind of bespoke aspects, whether that was tailoring a whole back office system, or building a REST API on top of some complex job system, we really kind of ended up running the gamut, in terms of the range of things we would take on some more successfully than others, inevitably, but it did become very varied. I would say,
Carlton Gibson 3:19
okay, okay. And so the, I wanted to ask you, then how you got into contributing Django because you've, you know, you've been, you're just very active, and you, you come in on particular issues and make very insightful comments. And as I look through the hash Issue History, you've been doing that for a long time.
Keryn Knight 3:38
Yeah, I can't remember when my first ticket got opened, it was probably around 2013 2014, when I actually found something that might be considered an issue. The longer I've worked with it, obviously, I've become more familiar with the kinds of things that are likely to be considered bugs and the kinds of things which are likely to be No, this is actually a problem in, in user land. I would say that, you know, one of the things is to just keep trying, fully 50% of my tickets over my career must have been closed as won't fix because they were some misunderstanding I had or something that had been retread elsewhere in a previous ticket or something, but it's a case of just continuing to kind of try your hand at it. It's not a difficult thing, particularly. And the most, that really happens if someone closes it as inevitably not relevant. That's the worst that happens. The best that happens is you either get a new feature that you really after, or you fix a bug that affects you and a bunch of other people. And I think how I ended up starting that contributors. Excuse me, contribution process was really just A couple of things I encountered that I thought were errors, and they weren't. They were minor errors, or they were a third party libraries errors. But it begins that process of actually talking with the community about how things progress and whether it's right that they're not in court or are in call. And it's, you know, it's a potluck for me sometimes as to whether it's, it's going to get accepted. And that's always interesting, if nothing else, because you can't have everything and but much as you'd like to, and there's obviously whole cohorts within the community that have divergent ideas about what should be in there and what shouldn't be in there. And they can become quite polarizing. So strike straddling that middle ground of what makes the cut is always an interesting thing to try and establish. Okay.
Carlton Gibson 6:02
One thing that you've quite often done, I've seen in recent discussions, which has been, you know, super handy is I know, we'll be looking at adjusting the settings object or something, and you'll come in with a way, you know, but if you use it in this particular way, which, for me, it's because sometimes quite a left field, right, like, you know, to manually be configuring, so calling settings configure with manual setting, that's something that, you know, I do, but not very often, and yet, you're quite frequently there with, ah, you know, if you configure a script this particular as you know, this happened. So,
Keryn Knight 6:33
yeah, I think that's just a case of having been around the block, so to speak, for so long that I've encountered an awful lot of not uncommon, but less frequent use cases, and tried so many various things, you know, there's this whole aspect of, of Django, and indeed, Python that I'm not as well versed in, as I might hope to be, the ORM has come on leaps and bounds over the years. And there's bits of that now that I've never used because of that. But there's also other things where I have used settings, not configure to do something outside of a normal Management Command. And I've had to become aware of the foibles of doing so. And because that's the less common approach, it's more valuable for me to step in, in those kinds of things where it hasn't necessarily been considered because it's off the beaten path. If it's if it's the if it's the common 80% use case, then everyone's already going to be thinking of those things. Anyway. So there's there's not much I'm going to say that's interesting or poignant towards it.
Carlton Gibson 7:44
Yeah, fair enough. Fair enough.
Will Vincent 7:45
I want to ask an open ended question, which is, so you've been working with Django for coming on 10 years? What how has it changed? I mean, just big picture, because a lot of people are new to Django, and they don't have that history. You know, just what have you seen that you like and dislike in terms of the change in that time,
Keryn Knight 8:05
I would say, I came in just after the magic removal brunch, which I think was nought point nine, six to one, something like that. So I've already missed the boat on probably the biggest change in Django is history. I think some of the kind of bigger ticket items that have really helped the overall community have been things like the bringing into a core of a migrations module, which was previously self and the new migrations module is more flexible, and also more robust. And that's certainly not to do disservice of south we all used it prior to the inbuilt migrations. And the splitting out of like the template template backends, so that you can use Jinja, or Mako, or whatever else you'd like to use. I think that was a that was a great step along with the introduction of app configs. I think by and large, those aren't necessarily used as much as people might have expected.
Will Vincent 9:14
Yeah, not much at all. I mean, for certainly for beginners. It's like, Yeah, cuz I cover it in my book, and I'm like, you can do all these things. But you, you know, you won't necessarily, like I don't see it used a lot at the, or, you know, more beginner level that I teach.
Keryn Knight 9:30
Yeah, and I think I think there's some reasons internal to Django that that's the case. And some reasons, depending on them being an external library that doesn't get the same focus in the documentation as the built in Django templating language, for example. And, of course, Jinja specifically being so similar in look and feel to Django. People don't necessarily know why they might want to reach for that. And the answer to that is usually just To share performance one because it compiles bytecode differently and magically and I don't understand it, but it's cool. But I think the app config ones specifically didn't happen in the way anyone foresaw. Because there was a long time where the concept of a default app config meant no one had to adapt. And I think the overall adaptation has now happened in Django itself to force that, and finally deprecate that. So I think we may now see more dependence on things like an app config that wasn't there before. But that also comes with its own slight tricky intricacies. But those those are the big ticket items I can think of in terms of what's actually kind of progressed. And I don't mean incrementally in terms of like, as I say, the the kind of modeling of database stuff that's improved leaps and bounds. The introduction of sub query syntax to actually manually build those was a pretty revolutionary thing, because it allowed a lot more flexibility in what you could do that wasn't magically done behind the scenes for you. But I don't know, I don't know enough about those because they keep improving incrementally. And I might be on the last long term support version, I don't necessarily see oh, okay, I can use this for that until I get around to the next one. So yeah, there's, there's bits I'm less familiar
Carlton Gibson 11:30
with, I think the whole expressions API stuff, and it's just phenomenally powerful. And the constraints that have come in, yes, yeah. And it's like a hat. It's just all the, the, the algorithm has become like, a well, there's, it's very sophisticated compared to what is in it's like, it's kind of hard to learn these things, because the docs can be a bit terse.
Keryn Knight 11:53
Yes, I think, because it's become more powerful, I think it has also become harder to learn. And that's, I think, true of Python, in general. But the more we can focus on what these improvements actually bring to the table, the more the users can get their head around it. But yes, the expressions API is exceptionally powerful. And I probably don't make as much use of it as I'd like to because I haven't fully thought through how to do it in that way. Because I have that long history of using it. I'm used to doing it a certain way to work around the fact that wasn't there. And now I can do it. But I'm not thinking about it. Because it's too new, so to speak, in terms of my overall trajectory, using it
Will Vincent 12:39
is not the curse of learning more.
Keryn Knight 12:42
Absolutely. It's yeah, it's in many ways, it's easier for a new learner to come on board and get used to the modern way of doing these things. Because by the time they are starting it, the incremental journey has already gone so far. But it also means that you could end up not out of date necessarily, but stuck in your ways.
Carlton Gibson 13:05
There's nothing wrong. That's fine. We defend that we defend that vigorously on this show.
Will Vincent 13:12
Well, I'm sorry, Carlton, you guys.
Carlton Gibson 13:14
Well, I was just gonna segue off. You mentioned that the new features of Python so while we were talking about new features of Django in chains, what do you what's your sort of take on the way pythons evolved, and
Keryn Knight 13:27
I think it has become a lot more complex, especially over the 3.0. To now release cycle where there's a lot of additional syntax, there's a lot of additional things that people are looking to add into the code base that makes sense from a maintenance perspective, which is always a great thing to have, but doesn't necessarily make sense in that first kind of almost prototypal phase where you're first building something out. And I'm thinking especially of type annotations. They've become very popular recently, through things like my Pai, and there's a couple of others whose names I've forgotten, that do type checking. Those add a huge amount of verbiage to the average Python project for the safety of making sure it's doing the right thing, but that's that's exceptionally difficult to reason about for a new learner. And there's also things like the walrus operator turning up, which I've used it more than I thought I would,
Carlton Gibson 14:43
which is good. Is it in Django the other day? That was like, it's
Keryn Knight 14:47
great. It's really good. You know, there's I've, there's use cases for it that I didn't think I'd have. But by the same token, it is the introduction of an Another type of declaring a variable, and now a new user has to learn that. So I think Python is in a tricky position overall in that it's trying to adapt and become more modern. But it may ultimately be leaving New users in the dust. As a result,
Carlton Gibson 15:20
I think it was David Beasley had this line that Python fits in your brain, maybe it's not his life, maybe it's, you know, just but one of the great things it fits isn't the right size for your head, and you can hold it, or
Keryn Knight 15:31
it certainly was, I don't know if it still is, especially with the kind of introduction, the whole async IO ecosystem. And I think that's somewhat fractured, which is always going to be a difficult one to deal with. But intrinsically through the kind of 2.0 to 3.0 release cycle, you could hold it in your head. And that became slightly truer again, when 3.3 came around. And we had things like Unicode literals, again, but it is still expanding. And there may come an inflection point where you've alienated your new audience. And you're then left with power users, I suppose we would be, but who knows all of that syntax and have internalized all of those things. It's a difficult boundary to straddle.
Will Vincent 16:23
The Django is I mean, specifically on async. Django is having it basically be like a feature flag where it's there, but it's not a default. And you can turn it ON OFF, which I guess for beginners would be nice if Python had that. But I also see why for maintenance and power users who dominate the discussion, they'd want to just have some of these things in there and not be a separate garden, within Django, Python.
Keryn Knight 16:49
Yeah, I think, ultimately, it would have been nice if we could instantly just flip a switch and become async. Io, all the way down the line. Because that would have made everyone's lives so much easier from, you know, the Django fellows, the people implementing these things all the way through the pipeline, and the actual users in terms of talking about the deployment strategy. And the deployment strategy for Python has always been a difficult one compared to say, PHP, or go where you're just compiling a static binary. And that's always been one of pythons problems. And I think that hasn't changed. Particularly, it's got better in some ways, it's got worse in some ways. Now we talk about the async. Io situation, and we talk about red and blue functions, and how those affects the overall ecosystem. And ultimately, it is still a transitionary period for Django, I, I know, at some point in the future, it will be entirely async IO all the way if you want it. And at some point in the future, it will become that by default. But it's still a process we're going through in terms of getting that adaptation all the way through the whole chain, which is, you know, substantial. It's a full stack framework for the back end. So there's a lot of moving parts. And there's a lot of things that can change at runtime, which makes it slightly complicated to even figure out if it is synchronous or asynchronous.
Will Vincent 18:27
Do you agree with that, Carlton? It's somewhere down the line?
Carlton Gibson 18:31
Oh, well, well, the end point. Will it? Uh, yeah, maybe I mean, in principle, why not, because in principle, it should be slightly less resource intensive to use asynchronous IO. And that there's a context switching cost. But, you know, if we can, if we can make it async, all the way down, then you should be able to declare a sync view at the top if you want to, I mean, why not? Asynchronous async Rhenus code is because it's more difficult to write. And it's more difficult to reason about. So 90% of view code that you write should you should just be using sync Django like you've always been using, because that's going to be the simplest to write the simplest to maintain. And it's going to be as performance as you need it. Then why do we need a sink was an asynchronous story? Well, Django is, you know, full stack web framework, it's there to have all the batteries and you don't want to change web framework just because you need high throughput for, you know, a particular endpoint or you you want to make some parallel API requests or something. I don't know what you know, but we need a story there. Do we need to be the fastest web framework in the block? No, we don't. So I don't know. I don't know what how where they end up I do see though, that you know, for instance, to get we've got an async view, dispatch, bam, thing in place now, but we still have to kind of wrap this the synchronous signals that, you know, request started requests finished. I still synchronous, so we have to wrap those in a sink to a sink. wrapper. And that goes out to a thread. And okay, there's a slight context cost there. Can we get rid of that? Yeah, there's a PR to get rid of that. But there's a performance overhead. It's really complex. And it's like, okay, then it takes a long time to sit down with these PRs and really work through exactly what's going on and exactly why it's slower or exactly why it's quicker, or exactly what the error conditions are. And so it's going to take a little while. I'll agree with that. But
Will Vincent 20:35
well, what you're saying about the evolution of Python is, that's pretty interesting to me. I mean, I've studied history in college. So I always like to look at past examples. I mean, there is a lifecycle to programming languages. And the fact that Python was a teaching language, and then kind of got adopted early by Google. And, you know, in many ways, right? The data science people made the shift from two to three happened, because the website that was kind of a mess, but we had all these new people come in. But that does seem a fundamental tension of for a language, right? It's like who drives to use its power users? You have more, I think, background than I do, I'm just trying to think if there's examples of other languages that have gone too far in the power user side, and lost people. I mean, I think PHP, PHP has gotten a lot better. Over time, I would, I would say that I'm sure.
Keryn Knight 21:23
I would think I left just before the PHP kind of grew up. You know, I left before it had a package manager and composer, I left before de facto standard frameworks came out in terms of like Laravel, and Symfony. Yeah. So they kind of revitalized it's kind of availability, in terms of doing big backend systems, I would say that PHP always had the benefit of being exceptionally easy to write and easy to run. And it mixed concerns within one file often, but that was fantastic, because it allowed people to actually see their results. It grew up with the advent of PHP, five through seven, it developed all of these extra features, taking them from other languages and other frameworks, other ecosystems. And I think it very much is in a good place. Now, it's, you know, it's reasonably performant, it's got a great story across any ecosystem, niche you might want to fill these days, I would think in terms of the kind of language familiarity I have, Pearl would probably be the one that I would think has sort of entered into the end of its lifetime. And that's not to say it's a bad language. And it's not, so people aren't still using it to do incredible things. It's a great text processing language. But it's one of those things where it's time in the sun has waxed and waned. Whilst Python is still kind of Currently, over the last 10 years, very much in that middle ground, Python has seen a huge increase, because of things like the machine learning and all of that kind of stuff that that I don't really understand AI, it's it's an entire part of the ecosystem that is beyond my understanding. But there's a lot of cool stuff happening with like pytorch and TensorFlow, and people are using it as a glue language. And that enables the pickup, where people don't necessarily have a full understanding of the language, they can still write the things they need to write. I think, you know, in the, in the long term over the next 10 years, we may find that some of that dissipates because those use cases will be fit by a different language. But we may also find that it just gets codified as Python is the language to do X, Y, or Zed because it's been doing it for so long, because it has a decent API for it, what have you. It's it's still, I think, too early to say that Python is on its way out in any fashion. I wouldn't. I wouldn't bet on that, particularly. But it's an interesting period. And it's become more interesting over the three point X release cycle with all the new stuff we've got as a result of that. And I can tell,
Carlton Gibson 24:28
you can still write it without all the, you know, type annotations and things you can do the you can bang out your 30 line pi script that does what you wanted to.
Keryn Knight 24:37
Yeah, I find I find I have a love hate relationship with type annotations in that. If I find encountering some gnarly bit of code, and it doesn't have them, I really wish I had. But I also don't necessarily want to write them because historically being normal typing, they've seemed like the wrong kind of fit for what I'd like to do. Python is a duck, duck typing based language. And more recently, we've got structural typing in terms of protocols. And I think that that's a great addition. But unfortunately, it's still relatively recent. So a lot of the type annotations you see out there will be field, this is an integer, this is a class of XY or Zed, where perhaps they could be more flexible. And I think that's, that's probably part of the reason that Django itself doesn't have type annotations. If you look at the various sub projects for it, they are exquisitely detailed, but exceptionally complex because of that duck typing, that means you can pass anything in that fits a certain structure. Yeah,
Carlton Gibson 25:50
but even if you just take to take the channels project, there was a PR to add type hints onto there. And just taking the example of a user, you know, where where a function takes a user object or Django user object, like the type hint was horrific. And because it can be an abstract user and abstract base user, you know, it's just too many types. And so you might create an alias where you take a user is, in fact this, this horrible thing, but it's like, wow, that's that really is quite flexible in what's accepted in this function.
Keryn Knight 26:24
Yeah. And I think that's one of pythons strengths, ultimately. Yeah. Yeah.
Carlton Gibson 26:29
Yeah.
Will Vincent 26:30
Well, I'm PhD PHP, I have a mate who runs an agency actually, though, he just joined a company full time, but he's uses Laravel. And so he's always forwarding me things about Laravel. And, and they've, there's huge advancements there. Sometimes I feel like the people I know in the Django world were a little bit myopic in terms of as fast API, but non Python language frameworks, like Laravel has a lot. That's interesting. But the model where it's just one person owns it and runs it. So they can just make decisions is sort of this balancing act. It's sometimes it's nice, they can just do it, and monetize. And they don't have a Django Software Foundation Board. But there's less of a community or it's a different type of community. So that's interesting for me to see, he's constantly sending me all these features that they can do and monetize. And sometimes I'm jealous that they can move faster. But I also think about what, you know, what's the cost? I mean, because it's a one to one man operation ultimately, still. And I wonder how sustainable that can be?
Keryn Knight 27:30
Yeah, I mean, I think it's, it's a difference in that community aspect. Because, absolutely. And obviously, I don't have a huge amount of experience within the Laravel community. As I said, that came much later. But having that strong personality at the top defining, this is what we're going to do. This is what's coming next is a great thing. And it's it allows you to push forward your state of the art in terms of what you're producing quite quickly. But you have to balance that with the amount of churn that produces. Django is an exceptionally old framework by modern standards. It's been around like 1516 years or something now at this point. So there's a lot of users, and a lot of us are used to doing things in a certain way. Certainly, there are things I wish Django had done differently. And or I wish Django would jump in and produce something that hits the 8% case that I see it's missing. But it's more democratic in that it's all just ultimately contributions from the community, putting them into track and the mailing list and talking about those things. And we have the Django enhancement proposals thing to make sure that if something big is coming into chords, wellspect out, and I don't I don't know what Laravel has in terms of that. But you can see that they are moving fast across a lot of different fronts. And it's an interestingly different community in that aspect. And yeah, absolutely. There's parts of that I'm super jealous of because they move fast. But there's also a part of me that's like, thank goodness, Django doesn't move fast. Yes, I might wish it had X, Y, or Zed functionality, or it done this differently. But it's a very stable framework overall, the number of times I've had real issues, changing things on an upgrade cycle, because the third party dependencies, most of the changes internal to Django, are one, they're in the release notes and well covered there. But additionally, they are relatively minor because it's got that incremental path, and it doesn't really subscribe to semantic versioning. So you just upgrade to the next one. The biggest changes when you go from long term support to long term support and even that's mostly just warnings. So it's a Yeah. I would say I wish there were things Django would do, and faster, but I entirely get why it doesn't. And certainly it hasn't stopped me using it over these last 10 years. So hopefully great effect.
Carlton Gibson 30:20
Okay, the traditional sort of answer there, though, is to do things as a third party package, you know, as an add on, do you think do you think there are limitations to that approach?
Keryn Knight 30:29
I think doing it as a third party add on is often the best case initially. That's certainly how migration started out. That's how static files started out. And there were various community provided ginger integrations, for example, to cover off the things we talked about earlier. I think the problem with relying so much on third parties is one you're then beholden to an even smaller cohort of maintainers. Because a lot of these third party packages, just one person won't have yeah, there's just one person. Absolutely. So So you become dependent on a smaller number of maintainers, who become too dependent on a larger divergent set of maintainers, who obviously have different priorities, and they have, you know, lives that they want to live independent of their open source contributions. And ultimately, whilst it does sometimes pave the way for things to come into Django proper, it does also mean that Django itself doesn't have an answer for that. And the biggest, you know, one of the one of the biggest areas now served by Django itself, I would say is, REST API is using REST framework. Previously, we had things like, tasty pie very early on. There was a couple of others, but rest framework seems to have taken the mindshare overall. And that's great, because it means the community works entirely within that ecosystem. But it's still a disappointment that when you come to say to a new user, let's use Django for this. And they ask about that story. How to rest API's work, what does it have there? The answer is it doesn't have that. I mean, it has it absolutely has the kind of underlying primitives. But it doesn't have the full story. So you have to reach out for that third package. And then you're into that kind of maintenance aspect that I talked about. So as with everything, it's it's a difficult balancing act. And the notion of increasing what's in core, I think, has been diminishing over time, because core has had some things ejected out of it. I think the there's a couple of little packages, form tools, I think, was one that got made into a separate package, because it's not the most common use case. But I think there are some things where Django is missing a story for the 80% in that 80 20% rule that it could be serving. And I don't think there's a good answer to that, because it's a lot of extra burden to take on for a very small kind of core set of people who are working on it day in day out. And it's a lot of extra burden. But when you evaluate it against saying to people, there's nothing built in for that, especially when the web has changed so much over these last 15 years. When when Tango first came onto the scene, it was revolutionary for the Python community in the same way Rails was for the Ruby community. And I think in some aspects, that's less true now. And you see that in the competing frameworks that are kind of capturing mindshare today, like fast API or black sheep, what have you. They black sheep. There's there's
Carlton Gibson 34:17
always another one. Yeah, this framework of the week.
Keryn Knight 34:21
Yeah, absolutely. And they are interesting, and they serve a different purpose to what the core of Django now provides. Django now provides a template language and always has, but that's not even necessarily making up the majority of use cases. It may still be I don't have any data on it. But certainly there's a lot of rest driven API's forgiven value for rest. And those aren't going to be using the kind of template language back end anywhere near as much so it's always a hard thing to balance that, and I don't have to sit in a position where I make the judgment, which is fantastic for me. I mean,
Carlton Gibson 35:06
just on the read the rest, rest framework type site, there's, there's a couple of tickets, one to add to extract the request passing interest into it kind of separate classes, we've got the multipart parser. So you know, form data comes in. But to make that turn a pluggable, so you could have a JSON parser in there, and then we can have content negotiation, and then at that point, it becomes okay, you could accept JSON requests and form encoded requests. And Okay, that looks starts to look more resti. And then, you know, the serializer story we don't quite have, but you know, there's no reason why that couldn't be bought in, and then you've got 80%, of what restaurant gives you with the, you know, the generic version.
Keryn Knight 35:48
And that's, that's very much something I've been thinking about recently, too. Because I think, as you say, half the story is already there. And the missing half is difficult to do in a backwards compatible way, but not insurmountable. As you say, the content negotiation, the pausing of the request body, these are all things that we can make work. And once you do, even the forms API, I think, could probably be adapted successfully. Because one thing people don't necessarily fully understand about the forms API, in terms of internally is it's a reasonably good API for validating a dictionary, which is perfect because most web requests these days are perhaps nested, but they're just dictionaries of here's some data. Now, the forms API itself doesn't currently allow you to have nested dictionaries. But there's no reason it couldn't. And I've been thinking about how I might tackle that. Because, yeah, I think once you have that, and you've got body pausing, and you've got content negotiation, you've got like three quarters of what people actually want. And the forms API then has enough rich metadata inside of it, that you could do stuff, like generate an open API schema for the whole thing. And you get, you know, a Rolling Stone effect, it'll have knock on effects for everything. Rest framework would be able to become lighter as a tasty pie. And so those extra dependencies you do have would have less surface area. And more eyes would be on those other bits, because they're now in Django proper. So that that's, you know, a promising thing.
Carlton Gibson 37:37
It's no Java question. Well, otherwise, I'm going to swing always, but
Will Vincent 37:40
you go ahead, but Well, okay.
Carlton Gibson 37:43
Well, if you've got a question cutting, because
Will Vincent 37:46
not as good as probably your question, so you go ahead.
Carlton Gibson 37:49
Okay. All right, fine. So scares, the other thing I wanted to really talk to you was about was, you've done lots of tickets on optimizing, you know, little bits of Django or on profiling bits on the template engine, you're working on speeding that up, you mentioned that earlier, that changes faster? Well, okay, let's make the detail faster. Can you still talk about you know, how you go about how you approach an optimization ticket, and you know, your thoughts, then the work the process, you go, because I find it really interesting, you open these really detailed tickets that spell out, you know, some real clear gains that then make it easy to review and easy to think, oh, yeah, let's get that in.
Keryn Knight 38:25
The process is exceptionally hit or miss the tickets you see that? I'm asking you to consider in terms of merging other ones that have proved successful. I've been down so many dead ends, that it's unreal. I have so many notes just floating around with numbers in that don't prove anything. Yeah,
Carlton Gibson 38:48
that's important. That's important to hear as well, though, because if people are thinking,
Keryn Knight 38:53
yeah, if you're looking at my tickets that are performance related recently, they probably represent maybe a 25% success rate in terms of things I've actually found. There's whole other, there's pages of notes around Okay, well, can I try and speed this up? Not targeting any specific area, but just a thing that looks like maybe I can speed it up? And the answer is? No, because because of consideration XY or Zed that I hadn't seen elsewhere? Or because I'm not super familiar with that bit of the API in the way it's used. And I have to wait 500 seconds for the entire test suite to fail, essentially, and say, No, that was wrong. So by no means is it 100% success rate. It's mostly a case of finding a section of the code that I think probably could have some speed up applied, and seeing if after that speed up is applied. Is it even worth it? Because again, Even if I make a change, and it's 50 nanoseconds faster on my machine, if it's in a piece of the framework, that's not a hot loop, essentially, it's a waste of time to really deal with it. If it's called once per process, and I'm saving 50 nanoseconds, it's probably not worth actually wasting the time on because everyone's already got a whole bunch of things to look at and work on. So it's got to be relatively frequently used, or a pathological case. And then it has to show actual meaningful value and deriving that meaningful value is, it's, it's difficult, because measuring things is not easy. I won't say I get it right 100% of the time. But if it looks repeatable across multiple runs, if it looks like I can come back and rerun it, in two days time, and it may not have the same numbers, but it's got the same kind of overall difference. That's when I consider opening a ticket. And so that that means a lot of data collection. And that means writing any individual ticket about it takes quite a lot of time, because I need to make sure I'm getting the data, right, and explaining why that data is relevant. And ultimately, all of that might be to save a couple of microseconds here or there. But it happens a lot. And so that build up allows it to actually have a real difference in, in production, so to speak. But it is exceptionally it is exceptionally hit and miss. I won't try and paint it as always 100% successful. No, no, no,
Will Vincent 41:46
I think testing is reminds me of there's the XKCD about compiling code where, you know, it's like the number one excuse of programmers goofing off, and they're fighting with swords. And the manager is like, what are you doing, like codes compiling? We don't really have that in Python. But the test suite? You know, it's,
Keryn Knight 42:01
we do when you run? Yeah, I mean, that's,
Will Vincent 42:04
and you know, I mean, we Adam Johnson, we've had on numerous times, he has a whole book on speeding up your Django tests, but in a larger code base. Yeah. If it takes more than 30 seconds, you know, it's like, well, that's a 15 minute break right there just every time. So,
Keryn Knight 42:18
yeah, I try and make sure I do as much work that I think will run successfully before running all the tests. As I say, I think it takes me about 500 seconds to run them without using the parallel runner. So it's a significant chunk, I'll go and have a coffee and see what I come back to. And if it's good news, maybe there's a ticket. If there's not, then I will hang my head in shame and move on to something else and see or see if there's another way to do it.
Will Vincent 42:53
I want to ask about Django technical errors, which is a new third party package you have up which has subjectively better for four or 500 pages, I was actually going to, I forget where I saw maybe I saw this just looking at you before this interview, but a couple weeks ago, I wanted to put in the Django newsletter. And Jeff Tripp was like, Oh, it seems kind of alpha. But I want to put a note being like still still Alpha. But that looks very cool. Like, can you talk about where that's at? Like, can I put it in the newsletter? Are you comfortable with a lot of people looking at it,
Keryn Knight 43:22
I mean, I would say I'm comfortable with people looking at it, I'm less comfortable with people using it. It's, it's still in that alpha stage of I'm still thinking through all of the ways I'd like to display the information and figuring out all of the different pieces of information that Django is existing error pages display. And the only reason I'm looking at it is because those error pages stayed substantially the same for most of Django is lifetime, but the web has moved on. We have better understandings of things like accessibility, we've got things like dark mode, people who have preferred reduced motion. All of these things mean that there's probably something to be done in terms of improving the error pages. And the reason I call it subjectively better is because I'm not as well versed in accessibility as I like to be. And it's a lot of experimentation that I'm doing there. Hopefully, you know, hopefully, one might be able to say they're objectively better, but I certainly wouldn't make that claim. Because I don't have a full scope of how it should be and how it actually works for the users. It's currently just me playing around with it. It's in a promising state. It's currently using tailwind to output all the CSS mostly as an exercise in getting familiar with that because that's relatively new, and I hadn't had a chance to play with that. particularly, but ultimately I would be ditching that in favor of actually just embedding all the CSS in the file. It's yeah, it's in progress. It's very much Alpha. I think it's cool. I'm kind of enthusiastic for it. But if nothing else, I'd like it to spark a conversation about improving the error messages, and all of that surrounding stuff, I don't necessarily think, as it is, it would ever end up in core, though, you know, we'd be thrilled to have that. But if, at least, if when I park it, it has achieved the job of someone better investigating and providing new error messages and new error pages, then that would be a job well done. And I've been doing it I've looked at a number of other error pages, whether they are kind of from competing frameworks, or what happens if you Google error page or debug page on Google images, and just seeing what other people display and how they might display it. So there's probably some aspects of it that are cribbed from somewhere else. But I don't know where. And it's just trying to make something in general. That bit easier to pause. And I think that was easier for the Django error pages under Django to when we didn't have stalking exceptions. Whereas now as an exception being displayed can be very long in terms of the individual frames, because zero division error was caused by Unicode error somewhere else. So yeah, I'm hopeful for it, but we'll see where it goes. I wouldn't recommend people use it. Yeah.
Carlton Gibson 46:49
It's a brave thing to take on the the error page, because they're kind of venerable. We had Karen Tracy on a few weeks ago. And she, her book was written like a decade ago, and that almost unchanged from from what she's.
Keryn Knight 47:03
So I've tried hard to point out and in the readme that is objectively better. But the current error pages have stood the test of time exceptionally well. I think, you know, anyone who comes back to Django, having five years off, can still read that same error page, it's not drifted by any substantial amount is just improved incrementally over that time. Which is fundamentally what Django does,
Carlton Gibson 47:30
is just, that's
Keryn Knight 47:31
Django in a nutshell. Yeah, incrementally improved. And I don't think that's a bad place to
Carlton Gibson 47:37
be fantastic.
Will Vincent 47:39
Well, I know we're coming up on time. I had one one more, but do you have something you want to imagine? No, no, that's why just gonna say, We want you saying about Python. And its evolution. I mean, we we the Django Software Foundation, think a lot about Django and how to make it secure. And I'm the treasurer. So I'm looking at the books and trying to think, How much money do we need, you know, we always talk about expanding things are keeping it small. You know, I would say big picture, pulling API's in at some point to core in some way. That's a no brainer, I think that will happen. We'll probably need to expand number of fellows. I mean, we we are we the DSF is lucky that Carlton and Maurice are so good and so efficient, that it's not an issue right now. But at some point, we'll have other fellows at some point, you might want to expand, we could always raise more money, if we needed to, to fund things. So these are kind of the questions from the DSF level, we think about, you know, how do we make Jango better, but I would say for beginners anyway, so my wish list, I would say, bringing API's in in some form, CMS, I mean, you know, Laravel has API's and CMS wagtail is amazing. It's kind of the default. Now. I don't know if that needs to be brought in the same way API's does. But then deployment deployments, the deployment, and maybe I should at some point, write a another official tutorial that kind of gets people kind of what I have in my books, but something that says here's how to write and launch a Django project for beginners, because that's, that's missing a bit now, partly because deployment is tricky. And also because subjectively like saying, well, what's the default? You can, you know, what are the settings configs? Like, how, what's good enough to put live? But I think those three things if there was more about CMS and the official docs, bring the API's in, and a way to build and deploy a project from the official docs. I think those would all help and certainly, some of those are doable. But I'm curious what your list is. I put put that out there is that's me, ruminating publicly.
Keryn Knight 49:47
I mean, the list of things I'd like to see changes on a daily basis depending on on what I'm dealing with, I suppose, but certainly having an answer for Building the majority of an API would be one of the top ones. I think it would be nice to have. And it's a risky proposition. But I think it would be nice to have an answer to how to do the front end ecosystem. There's there's a number of competing packages in terms of how do I bolt Webpack onto it? How do I make use of Babel or TypeScript? Because the front end has become such a separate beast than it was previously. And it's got its own ecosystem and its own configuration to deal with. That's still a difficult thing to get your head around if you're not fully deep within it. Now, I think Rails has had more attempts at doing that. I think it has sprockets in the asset pipeline and stuff. And they change things more frequently, as far as I'm aware, and the front end ecosystem has moved at a very quick pace over these last five or six years ultimately. So adding something like that interCall would be a risky proposition because you may end up switching out two versions later for the current in the thing. But perhaps that's settling down with things like as build and SW see that allow you to kind of make use of JavaScript modules or tailwind packages. So maybe that is becoming something where you can finally add something in and not just risk it needing tearing out into versions time. I think that would be a good thing to have an answer for. And I think at the moment, it's fine that the answer is use this popular third party package because it's so changeable. But I think one day perhaps when when it all settles down, and things have stabilized fully. If it ever happens, that would be a cool thing to have.
Carlton Gibson 52:08
I've been waiting for JavaScript to sell.
Will Vincent 52:12
htm X, have you used that? Do you have thoughts on that?
Keryn Knight 52:15
I haven't used htm X. I have used the previous version of Carson's intercooler intercooler. Yes, I use that on a number of projects. And I think I have a package for that somewhere as well. Just to kind of make the Django side of it more useful and more built in. I really liked intercooler. And I think, whilst HTML isn't a big change from what intercooler was, I think it's happening at the right time, in terms of, he seems to be having much more success. Finding an audience for htm X than intercooler, which was always possibly a little too early for its time it was, you know, competing with Angular and Ember and react as the things that had the zeitgeist at the time. And now, I think we're not necessarily seeing a push back on that. But with things like HTML, and alpine J, s, and then you've got the various live view things that Phoenix has, and I think Laravel has got an equivalent, you're seeing more decoupling again. So where we might have had a full offline first react front end, some people are pushing back into, we can do a lot of it on the server and just add in that interactivity we need via this small package. And I think that scales relatively well, your your system has to be very interactive or very complex, to not be able to get by, at the very least with some combination of the two. And I think most, most people with a middling amount of development experience of something of that ilk would know when to stop doing that and move to a fully separate front end and fully separate back end or merge the two completely. And that's always a different inflection point, because it depends on the project depends on what it's doing. Is it so interactive, that it's WebSockets and streaming diffs all the way down? Or is it just an occasional Ajax update to refresh some chart or other but I think it's a super cool project. And whilst I've not used it, having to use the previous one to do a couple of projects, I'm really enthused by that one.
Will Vincent 54:36
Yeah, it feels like it's it's moving up the that gap in terms of having to switch over because like, I'm just about to well, by the time this comes out my updated API's Django API's book will be out. A lot of newcomers come to me and say, I have to use API's and react or view and just every, you know, bag of tricks. And I always say, Well, you incrementally get there, you know Start with start with just Jango do stuff in the templates only switch to API's later unless you're experienced or know you need it. And so HTML is just keeps pushing that out, which certainly for people learning or on smaller projects, if you don't really need it, it's a whole other world to manage. So I think that's, that's good for all of web, but especially for Django that helps. There's more you can do with just default Django.
Keryn Knight 55:22
Yeah, there's a simplicity to it. That's powerful enough to get a lot of stuff done. But also means you are just progressively enhancing progressive. That's the word Yeah. And yeah, I think there's there's a lot of value in that. And I think dovetails quite nicely with the way Django works in that Django has always been about getting something that works, and works well. And HTML X, and things of that ilk that are small, but powerful, allow you to do that on the front end as well. And that's certainly not to criticize react or anything, you know. But yeah, I'm a big fan of the kind of progressive enhancement route.
Will Vincent 56:04
Carlton, you had? Well,
Carlton Gibson 56:06
no, I mean, I totally agree with all of that, obviously, it's the I think, as well, the, you know, you talked earlier about extending the forms API maybe into, you know, arrestee, serializer. Type domain. And what I find is really nice with HTML is the ability to push, oh, I'm still using form, oh, I'm still using form, oh, I'm still using a form. And you get to the, you know, maybe with a sprinkle of Alpine. And instead of, instead of sending JSON data, you just send form data, you know, the JavaScript got the form object, it's like, oh, I'm still using a form. You know, it's like, I haven't actually got to the point where I needed JSON at all yet. And then, you know, if we can then just get the forms that one step further out such that they can sensibly interact with JSON in and out, then actually, it's like, oh, wow, the surface area of what we're able to cover with the core, traditional Django stuff just grows to cover the 80%. I think I think it does cover most key use cases at that point, then yeah, okay. Yeah, a specialist endpoint, you know, blah, blah, blah. But that, that's what I mean, excited by it. I think it it knows me to do really quite straightforward, Django for much longer than I've been able to do for you, like literally half a dozen use of Oh, no, now you've got to go jump into this front end world. And it's straight to the API. No, I wouldn't, why would I go straight to the AI API, it's, it's really changed. The phenomenology of building a website, you know, in a way that I find very attractive. So yeah,
Keryn Knight 57:45
I agree entirely, having having been around for long enough that that's where I started doing it. And then there's that middle ground of everything becoming an API. And now we're maybe seeing it go back the other way slightly. There was a simplicity to it. And whilst you know, a lot of users now expect more interactivity and expect all of the extra stuff that the front end can do that it previously couldn't. There is there is that successful middle ground, where Yeah, it's still just a form coming back. And it's just augmented with the validation error of this field. And that validation is correct for their current content and wedge, and it's in the right position, and it hasn't had to be duplicated in the front end. There's there's a simplicity to that. And it's it's really nice to see that kind of thing come back. And I think for Django, specifically, that's been helped somewhat by the whole forms API becoming template based, where previously it was all just strings. Being all strings made it difficult to add classes, it made it difficult to add these extra data attributes that Alpine and HD MX want to add onto things. Whereas now with the forms API fully within the template system, it's relatively easy to always augment it with this tailwind class, this bootstrap class, this htm X attribute, this Alpine x data. And so it's allowed it to become more seamless than it otherwise would have been. I don't think I'd have enjoyed working on a project that used alpine or h2 mix or intercooler. If I had to manually output all of things, or augment them all in the Python widgets themselves, so I think I think that's dovetailed quite nicely.
Will Vincent 59:41
Well, just as there is the jam stack in JavaScript, maybe we can get the damn stack going, you know, Django and API's and markup because we've Yeah, let's let's get our cool. Yeah, well, we had here in the US we recently there was a postal service put up a site to so everyone could get COVID tests, you know, Two years late, but it was a static site. And they just had, I think it was jQuery or something for the forums and connect to a back end. And they put it together quickly. And it worked. And it scaled to, you know, 350 million people or whatever. And a lot of people were like, wow, that's, that's amazing. You can do that. And it's like, yeah, you can do that. Like that's, it doesn't need to be this big piece. They had to like subdomain, the US Postal Service website, because it's its own thing. But it's like, yeah, you can be simple and scale. And it doesn't need to be, you know, because everything is, so much of the web is just a form. And,
Keryn Knight 1:00:33
yeah, I think there's there's a tendency, because we're all driven by interesting, new things, there's a tendency to add complexity, for the sake of either learning it or because it's the thing to do. And yeah, I think sometimes the performance and scalability thing falls by the wayside. You know, Django is not the most scalable, excuse me, not the most performant thing in the world, because it's backed by Python, Python is getting faster. So Django will get faster. Cool. That's fantastic. But it's never going to compete with something written in C++ or something. But it doesn't need to. Because the actual performance and scalability for a lot of the web is, as you say, just some form over a database or over an external API. And so much of it is data entry and in some fashion that as long as you can horizontally scale up reasonably well, your big bottlenecks, probably still going to be your database.
Will Vincent 1:01:44
Yeah. Well, that's why that is the last thing, crud, the concept of crud, for beginners new to web development, when I tell them and show them that everything is 99% of the web is crud. That's a real aha moment for people and they realize, oh, Facebook, crud, Twitter, crud, like every site, it helps kind of piece together and provide a context that newcomers, you know, you don't think about if you're just using the web. And once you get there, it's like, oh, it's not that hard to create these sites, because you're doing 90% of it is the same. You're just changing the skin. And there's a little bit different, but you know, that's something I enjoy showing people and you know, it takes sort of five or six, but when they see oh, wow, like, it's all kind of the same. It's like, exactly like, that's what I'm trying to get you to it's not overwhelming, like there are these niches and deep areas, but it's kind of the same thing over and over again. But you don't until you build it, you don't think about it.
Keryn Knight 1:02:37
And that's that's pretty fundamental to the web. It's so you know, exclude exclude things like web sockets and all of that. The web's firms are just other words for crud. Possibly, they're better words than crud because they don't form a bad sounding acronym, but the acronym is at least catchy that once you hear it, you understand what it means.
Carlton Gibson 1:03:02
Fair enough. Right? Let's call it I've read it. I've hit up on our so Kay, thank you for coming on. And thank you for all that you do for Django. It's you know, your silent hero, but you know, thank you.
Keryn Knight 1:03:15
Anyway, you call you call me or say silent hero. You're the silent hero. You do all of it. Yeah, no think I don't say it.
Carlton Gibson 1:03:23
No, it's Mariusz. Mariusz. Anyway, right, that was Django Chat. Thanks for joining us, folks. Join us again next time. We're Djangochat.com and chatDjango on Twitter. Take care. Bye.