Django Chat

Django Security - Markus Holtermann

Episode Summary

Markus is a longtime Django contributor. We discuss the work of the Security and Ops Teams, his day job at Crate.io, async, and the future of Django.

Episode Notes

SHAMELESS PLUGS

Episode Transcription

Carlton Gibson  0:06  

Hi, and welcome to another episode of Django Chat, a weekly podcast on the Django web framework. I'm Carlton Gibson joined as ever by Will Vincent Halliwell. Hi, Carlton. And I will and with us this week, we've got Marcus Holtermann, who's a member of the Django technical board and on the Django security team, and you know, on the Django ops team, he's just a stalwart of the community. Hello, Markus.

 

Markus Holtermann  0:28  

Hi, Carlton. Hi. Well,

 

Carlton Gibson  0:30  

hi, everybody. Hello, thanks for coming on Marcus. So I guess let's, let's kick off. What How? How perhaps you could tell us a little bit about yourself. Marcus, like how did you get into program How do you find Django? You know, how long have you been with the community these kind of, you know,

 

Markus Holtermann  0:44  

I've been writing code for over two decades now. It's been it's been quite a while and then got into Django 2010 ish. As part of the German public to users dot t community. That's a Django project that started even before the days of Django 1.0 or something like 9.9 T or something like that very, very early stages of Django. That's when that project was started. And it's a support forum for the gems became open to the community. And yeah, features a bulletin boards, news articles and plan planets and all these kinds of things. Is that still going, it's still going, it's still maintained by a few folks on all. It's not unfortunately not open source. It's a kind of a thing that we since ever talk about, yes, we want to make this open source. But because it has so much legacy, it's also one of the things where there's a lot of technical depth in there that we want to have it the folks who work on it these days want to get out of it before they actually open source it. We are going to take

 

Will Vincent  2:01  

Is that the word that was coming to mind? Yeah,

 

Markus Holtermann  2:02  

probably. I

 

Will Vincent  2:06  

guess it's that's the kind of the word to start with. cr Yeah.

 

Markus Holtermann  2:10  

Yes, exactly.

 

Carlton Gibson  2:12  

But that's it. That's, that's just any old codebase. Right. any old like, so that's, that's what 10 years old. 15 years old. You know, that's an old COBOL?

 

Markus Holtermann  2:22  

Yes, it's I think they started writing that. I wasn't on the original team, but it's they started writing in 2006. Everything. I mean if and it's if it's a project where then as a side effects like Vax, like flask Jinja, two, all those projects came out of Yeah, it's, it's kind of in the early stages of the Python web part, but I guess the Django ish style web development part.

 

Carlton Gibson  2:56  

That's kind of cool. And that got you into Django

 

Markus Holtermann  2:59  

and then eventually got To Django, and yeah, there's Apollo 13 is on the team as well. I was back in the day was another theme as well. And then he eventually kind of like always pushed me a bit to maybe contribute to Django, because at that point, he was a core contributor as well. And, yeah, that eventually got me into contributing to Django after Andrew Godwin merged his migration batch in 2014. I believe 13 1414 1.7 1.7. Exactly. And yeah, then Ana contributors, back fixes now. Every other day, you want to

 

Carlton Gibson  3:44  

because you're late. So when tickets come in on the migrations, I always went a little bit because they're like that quite hard. And I'm always quite glad if Mark has comments or you know, a couple of other people. Marcus is one of my core, you know, people go to people on the migrations framework, and yeah,

 

Markus Holtermann  3:59  

man Still, I unfortunately hadn't didn't at the time for the last few years to actually do a lot of things on the migration framework. But I think I, it seems there's a bunch of other contributors, these days that actually have picked up behind me and behind Andrew and others that contributed on a regular basis and like, squash old, all these facts these days. Yeah. So there you go. Well,

 

Will Vincent  4:27  

I was gonna say that, I'm gonna put a link to all the various teams on the Django project. site, because I think you have the record for being on like, all of them, right. So there's, and again, and this is also in flux, you're on the ops team. You're, I guess you're not technically a release or you're on the security team. You're on the technical board. They're not the advisory board. So I guess you're and then you're on the technical team. So you're missing two but all the other boards and then that it all is in kind of flux because now there's the new Yeah, technical software. Perhaps what is the future gonna look like in terms of the organization? Right? Because like the term Django core, even Yes, tossed around. What do you make of all this for a casual listener? So how should they understand?

 

Markus Holtermann  5:12  

So about a month ago or something, beginning, first quarter, first half of March or something, we passed the Django enhancement, proposal number 10, which essentially states that we want to change how Django is being developed or how the Django community works, and how decisions are made and who has I don't want to say the right to do something but who has who can eventually if there's a discussion that doesn't come to the solution, where people can't agree on one or two, one or two things, and who can make the eventual or final decision to go and check. Okay, let's go with this way or another one. And Yeah, I guess that's that's one of the things that's, that's been one of the things that's been around and where we've been trying to do that for quite a while, but never really got around that now. James Bennett finally managed to actually write down the step. And the while at that point, the Django core team voted on that. And it was accepted with quite a large number of, of votes. And yeah, so then the future of the Django project is going to look different look, I guess. We're trying to open up the decision making processes and make this more visible.

 

Will Vincent  6:45  

Yeah, because the because the other piece was the so depth 10 and then the Corps voted on it. And then the the Django Software Foundation Board, which I'm on most recent meeting, we approved it. So going forward, I think the idea is that there'll be The technical board. And then the one I'm on, which is sort of the non technical things as kind of the arbiters of disputes or decisions that arise, because the next step is an election. The they have that Frank Wiles, who's the president is going to be configuring, setting up the election of technical board members.

 

Markus Holtermann  7:23  

Frankly, I'm not entirely sure about the exact processes from the, from the top of my head, because it's a fairly lengthy document. But yeah, we need to Yeah,

 

Will Vincent  7:32  

well, I mean, I'm privy to it is a long document. And James has spent a ton of work on it. Basically, the reason why Frank is going to take that on is because we don't want someone on the technical board to be in charge of the forum for voting, but it's just going to be a Google form with the proper permissions. Yeah. And seems like that will be a good idea happening. So but I think Part Part of that is, as you were saying is public versus private discussion. Right, that's sort of a big thing about. So what are some examples of why that matters for the Django community? I think

 

Markus Holtermann  8:06  

it's it's danglars, an open source project. And we should be able to make decisions in a public forum and make it visible to anybody who wants to contribute, to understand the reasoning of how the people who made a decision came to that decision, and seems to be just in the nature of how to do those things. And that making those decisions in public seems to be the right choice.

 

Will Vincent  8:32  

I think the other part as well is that the term Django core is something that you get, and you never lose, even if you're not as active. I mean, because for example, you're incredibly active, and there are members who have the title who are not as active. So I believe one of the intents is to, I think there that's going to be a legacy thing that can be bestowed but but to have the technical team reflect the current active members as opposed to people who were maybe more active in the past

 

Markus Holtermann  8:57  

all the time. Django core, or other Django committer Django core team member has been around is a term for somebody who had been given commit access in the past. And it was always kind of a. Once you have it, you don't lose it unless you mess up like big time. But I don't think anybody ever did or you handed or give it up by yourself. But then this is Yeah, as you said, If you then don't actively participate in the whole development of that, then you still seem to be on this list of people who aren't in there and you can still take credit for for that position. And sure, when you have contributed in a past significant amount of work, that was usually the thing that's kind of led somebody to become a core core member, then it's using that positional using the titles that you had in the past kind of made sense. seems fine to me. But over the last years, it's been clear that there's not really like this big thing in Django that somebody could do to become a core committer or core member. And the things that in the past indicated somebody becoming committed is just not really there anymore. So finding a way to become a core committer wasn't just something that still existed. There's there wasn't a route to become one. So dropping this whole concept and acknowledging people who had been at this position in the past, but then essentially dropping this whole position. seemed kind of like a better, good, good way forward.

 

Will Vincent  10:48  

One part of that is that because the term core was around before there were Django fellows because Tim Graham and now Carlton, and Mario's in terms of releases, at least they do much of that work. Work for the community. Yes.

 

Markus Holtermann  11:02  

So that's part of the exact day. So in the past, we

 

would commit access, you were able to commit to GitHub direct repo repository directly. And some people a small number, I think, like three of three people had released access, which meant they had the possibility to put packages in pi pi. And with the, with this with the new change of the depth 10, this also changed in a way that's you don't need to be, are you? Yeah, we will have a merchant role, which is the people who have committed access to the Django repository. And we have a release role, which are the people who have access to pi pi. And can put pages on pi pi. People can be in both but they don't have to be. So if you folks who don't have the time to constantly review in our time to merge patches. And specifically review is not necessarily up to the murderers. They should. Yeah, they should be doing like a Senate, like a check on style and so on. And do like general review under PR that just makes them whatever is committed or whatever they commit, make sense. But did this happen if the approaches taken in a PR should be taken by the community and by people who are going to be involved in this, this change? you could you could imagine, like,

 

Carlton Gibson  12:43  

you know, a number of people reviewing the PR and then the merger is just a person who clicks the button at the end. It's it's kind of the, you know, I always describe the fellow role as like the the janitors floor. That's kind of what we do is we just keep it tidy. I mean, we do do really viewing and

 

Markus Holtermann  13:00  

yeah, I mean, you do more than just being the janitor. You do far more than just being the janitor and for the Jenga project, but yeah, it seems to be a significant role to,

 

Will Vincent  13:12  

to what you're doing? Well, I think part of this in general is the reason why that like for me on the on the board and you and the technical board, why we spend time on this, because it's an open source project. Basically, everyone's a volunteer, and we wanted the system to be in place so people can contribute, but not one person has access to everything, for example. And so when people do leave, there's a process, as you said, for new people to come in to be a handoff of, of knowledge and power. I mean, even for example, so we added GitHub sponsors to a to a Jenga repo. And that's something I worked a lot on. So I have access to part of the Django repo but I couldn't fill out the form so I had to get Mario's who's the fellow because he has you know, more access. So these layers of permission are important just in case Yeah, you know, you don't want it someone to have the keys to everything and you want to have a bit of structure but not so much that it's prohibitive to get anything done. Yeah. This is how I think about this

 

Carlton Gibson  14:10  

is, I mean, the things particularly with access to like pipey I, you know, that's, that's sensitive, you have a package being uploaded as Django, two pi pi. If it isn't Django. Yes. There's too many people depend on it being Django for that to happen.

 

Markus Holtermann  14:29  

Exactly. This.

 

Opening up the whole thing, and opening up the whole contribution protests to Django and allowing more or less arbitrary people to be to be voted on to be murderers, while releases, comes with the site notes that we with a perspective and me taking this from a meta perspective on this more security, conscious person right now. Comes with the problem that we need to trusted person to not put in some make backdoor or whatever in that release package that somebody needs to inspect and and find and so yeah it's the the selection process or the vote voting process on who becomes them will be open according to the things that are stated in the DEP. But it's definitely got also going to be something where the community needs to look out for it's not some random malicious person is going to like hijack Django because that's not gonna be a good

 

Carlton Gibson  15:46  

but i was i was reading the the DEP the yesterday or day before some for some reason I needed to refresh my memory on it. But the the qualification to be on to the technical board have to Okay, the nomination of mergers and releases for these important roles, but to be a candidate for the technical board, so the political board will be elected by the DSF members. So already to be in the DSM, you have to be known by the community, right? Anyone can join, but you have to be known by the community.

 

Will Vincent  16:16  

Well, you can, you have to be you have to be nominated and then approved by the

 

Carlton Gibson  16:22  

candidate that

 

Will Vincent  16:24  

you can self nominate. But there's an actually I'm linking I wrote a blog post just recently about what the board actually does. There's I think, 180 or so individual members, so sorry, continue, Carlton, but that's like the first step of Yes,

 

Carlton Gibson  16:36  

but like that to be a tech so but not anyone can be on the technical board, you have to have a show, like a recent like within the last couple of years shown history of contributing and reviewing prs or working on track or that, you know, harp or B, being in the group. So, to be a candidate for the technical board is actually quite tough. It's it's not like Any any person who isn't known to the community and known to be a contributor could could even stand for the technical boards? I think so I think that the term the wording in the DEP is it's actually very conscious of the debt of the possible dangers if it were more open. So I'm quite relaxed about it in that, yeah,

 

Markus Holtermann  17:17  

it definitely, James definitely put a lot of thought into that part and in the other person has contributed in the, in the writing of the depth. I mean, James didn't major part of that. But again, the discussion on the depth was held in public on the, on the depth itself on the PR, and it was over a year in

 

Carlton Gibson  17:40  

discussion, right, it was,

 

Markus Holtermann  17:41  

it took a long time, and now it's taking quite a while.

 

Carlton Gibson  17:47  

But that's cool.

 

Will Vincent  17:47  

So maybe we can talk about some of these hats that you wear, because I hope that there's more of an understanding of the you know, the many, many roles that you you play in the community. So let's pick security team. So you and Carlton are both in the security team. And I think, what's the process for a normal security release? And then there was that the GitHub SQL injection injection one recently, which I believe you did the commit for that you you all turned around in two days, a month or two ago. Right. So that was first discovered on GitHub. So what's the normal process? And then I thought that that one was particularly impressive, where maybe it was three days, but then it was discovered. The commit was done, and it was released. Yeah. And

 

that was a usual I thought that was cool. Thank you.

 

Markus Holtermann  18:36  

And the usual process for security thing is usually that well, somebody reports either on hacker one on a security mailing list or a security reporting list. Security, a Django project comm reports a security issue or a suspected security issue, like rather be conscious, like, I think this might be an issue reported and then we go back Actually, that's not but that's the We've much rather have that than somebody just putting something out into the open like, Oh shit, this is something that we should have, like, fixed now. So usually you report something in a private way. And then people on the security team are going to review that is an issue. Is it like? Do we need to fix it like now? Or do we have time to, to rethink the whole approach we have there. We then usually talk to the reporter and try to figure out who's going to write the patch who's going to reveal the patch like be tritely be loved for reporters to also contribute to patch and then the people on the security team reviewing it. But if they feel like yeah, nuts, I don't even feel like writing the patch. Then somebody on the team is going to write a patch and we then hand out the patch or the proposed patch to the reporter and ask them to apply that and see that this actually fixes that bug. They're, well probably production environment that they have there. This may be goes back and forth. Once we are done with that we pre notify a set of organizations, companies have an upcoming security list security release. And we also pre notify the on the Django finance mailing list, that there will be a security release on a given date and time. And then at that point, well usually called an on or or mommies are going to commit the patches to the Jenga repository and issue the corresponding releases and another release. Now, that's, yeah, go ahead.

 

Carlton Gibson  20:46  

Go I was just gonna say about like that commits, because it's, like, potentially sensitive and on that morning of the release, and so we've got a time and it's like, we've got the patches ready and they say we're going to have to, you know, we just we just dropped one point 11 From extended support, but at the time we were supporting Django one point 11, Django 2.2 Django 3.0. And you have to merge those patches on to onto the branches and then within a short period, get the release out and the announcement out because once you've merged them, that those, those patches are public and so that it's kind of like, you have to get all the pieces lined up and it's a little bit tense. You know, it's it's much harder than a normal release, which you can you know, you take your time off. Yeah, Carol, Mark, as you were saying, so we've

 

Markus Holtermann  21:31  

Yeah, this is I can definitely understand that. Yeah, issuing a security release is a whole different story just from the mental capacity that's involved here because you can't like a usual release. You can go like, Oh, yeah, we broke whatever in there. You just gonna, it just don't install the new version, like pin something to before. And then like take the time to to fix this Balkans or reverted in Issue number one. But if that happens as a security release, you go like, Oh, damn, well, we just published a security issue, which we don't have a working, patched version for. And you go like, this is not ideal.

 

So they the issue that you brought up with the method is with a GitHub sequel engine wasn't. And he was like, this was an account hijack.

 

Will Vincent  22:27  

Right? Yes. Yeah. I was like the password. Yeah, that's what password.

 

Markus Holtermann  22:31  

Yeah, something like that. Um,

 

Will Vincent  22:33  

it was like a Unicode character or something. Yeah, you could shove something in there.

 

Markus Holtermann  22:37  

Yes.

 

Will Vincent  22:39  

But like you guys actually worked out. I'm just sitting here. Like,

 

Carlton Gibson  22:43  

the issue was that if you get a domain with Unicode characters in which sort of lower case comparisons to you know, certain database backends to write another domain, so you know, it could be google.com. But the the O's are funny Unicode characters, and then it would look like it was a Google image. But in fact, it wasn't wasn't a glimmer.

 

Markus Holtermann  23:04  

Yeah, in that case, like the issue itself is known, then you don't really need to treat it as a sensitive or private issue anymore. Because the moment those security issues unknown, impossibly exploits unknown. A bunch of smart people are going to look through all the cool places out there and trying to find projects. That's our code bases that are vulnerable. And at that point, this time between announcement and pre notification is just kind of wasted time. Because we should and stats half that time people already fixing people fixing our

 

Carlton Gibson  23:42  

installations. Yeah, so normally we give seven days but you can't give seven days when there's a

 

Markus Holtermann  23:47  

vendor out in the wild. So yeah, that's good. That's good. In this case, we just dropped the pre notification recently be issued a release or make a note for about a day or

 

something like that.

 

Will Vincent  24:00  

I was just going to say for Carlton, I often beat the drum of keep your Django version up to date. And even if you're using an LTS, you should do the minor releases the security releases for this reason, because otherwise you won't get. Yeah, the security updates. I'll give

 

Carlton Gibson  24:14  

you if you if you're on a version that's in the extended support, then it's definitely a, you know, visit. Point release. You definitely want that like,

 

Markus Holtermann  24:23  

yes, this

 

Will Vincent  24:24  

Yeah, cuz I think

 

Carlton Gibson  24:26  

it's security. You want that?

 

Will Vincent  24:27  

Yeah, absolutely. Let's go down the list. ops team. So you and I have had some, I've been emailing around that maybe we don't need to discuss some of that. But what is the ops team do like what is the ops infrastructure of Django itself? So you're on there along with Florian, Tobias and Carlton, um,

 

Markus Holtermann  24:47  

well, we have this website, and you probably have heard of it Django project.com. And that needs to run somewhere. And well, we also have a CI continuous integration that we use For testing Jango, which is run on Jing Django, ci.com. And hands odd when you run websites, you need some servers or something in the back end, that does work. And yeah, we use the pain of maintained it's the software or the not not necessarily software there, but the infrastructure. Like the upkeep thing. Jenkins up to date, usually we, when this fixes to the Django website, we deploy them.

 

These kinds of things.

 

Will Vincent  25:35  

Yeah. Well, it's, it's, I think, another sort of unsung role in a broader community, but quite a bit of work is done. And so for me on the board, what's as the treasurer is what's what's relevant is there's some costs involved with that. So I'm, monitor and help pay those. Rackspace has been sort of a major contributor the last few years, who provides the servers. They've been very generous with that. So just another one of these things. That's happening in the background that makes Django happen. And, you know, you, Marcus are very involved with. So I want to call that out. Thank you. I have a long list of things to ask you about, but maybe let's talk about your day job right? You work at crate IO, which is IoT scale database. Can you tell us all that? Yeah, yeah. IoT scale database. Explain that. That's not you. That's that's the headline on the website. Yeah.

 

Markus Holtermann  26:24  

The website which is fine. And so we build the database for the IoT market. So IoT is usually timestamp time series database, time series data. So something numbers usually or maybe some add a label to that and then a timestamp. And while you want to, like aggregate data from usually or in the market we are focusing on is the industrial IoT. So you have factories that companies have where they equip, they are production lines with sensors, all the machines themselves are already equipped with sensors, and qivana do analytics on on those machines. And you want to understand when something goes wrong. And like this whole, in order to do analytics, you need to store the data somewhere that then aggregates things. And turns out when you have a lot of plants or manufacturing lines and machines, and a lot of data and a lot of numbers, then you need to have a database that can cope with that. And yeah, with current dB, we have a database that's based on Lucene. Scales fairly well. Like pretty, pretty much almost linearly for I don't know even know how many nodes and can cope with ridiculous numbers of records in tables. And, yeah, it's a SQL database after all, so you still write your SQL statements and It returns Yana records.

 

Will Vincent  28:02  

So why not Postgres? Right? Obviously, there's a lot more that Postgres can't do, which is why he worked there, and you have this product. But what is where does if I start a Django project, that's IoT, and I'm using Postgres? Where do I see that? Oh, I need something better or bigger, like crate, like, do you have a sense of where that friction comes up?

 

Markus Holtermann  28:22  

So one of the things with Postgres is that it's transactional, for example, right? I'm not even sure if you can run Postgres without transactions, I suppose you can't, I guess you can't. You probably also don't want to, that's it. creatively, on the other hand, is eventually consistent. And you have kind of multi master image or you don't really have a master node in the in your cluster, you have a single node that holds at the particular mods set off target at the current time holds an information about the whole But that's known is also known to everybody every other node. But you can write to every node in your cluster connect to everyone, every notes write to every node read from every note. And it's more or less self balancing with the data. If you don't screw up your table creation, and you can easily go out

 

Carlton Gibson  29:21  

there what is the advantage of that is that engine speed you can pull in the rate at which you can ingest data is just

 

Markus Holtermann  29:27  

Yes, it's an order of magnitude higher. It's it's orders of magnitude higher. I don't even know the exact numbers, but we are currently looking at some of our bench markings. But it's sure you can scale your Postgres ridiculously high and improve a whole bunch of inserts. But at some point, it's going to, you're going to run into issues. And then for the most part in create, you just add two more nodes to your cluster to just increase Because it's like just in it, it's never going to be. You can simply again, in quotes,

 

Will Vincent  30:07  

but that's like tutorials, I always try to put simply just simply install Django. Because over Well,

 

Markus Holtermann  30:13  

um, but yeah, for the most part, it's, you add a note or two or five, and your cluster has more storage space, your cluster has more throughput, aggregation capabilities, memory, all these things.

 

Carlton Gibson  30:32  

And so the other question you said, eventually consistency, so So how long does that tape because I've used eventual ventually consistent data source and quite often that just means where's my data?

 

Right.

 

Get to get it back. It's like not found.

 

And you're like, hang on,

 

I just inserted it and you said it worked. Where is it? So that I mean, how long does that take to run through save your you know, I mean, I guess it depends on the size of the classroom, though, but

 

Markus Holtermann  30:56  

as I think it's sub millisecond minutes, area, like, you're not gonna wait five minutes for this number to show up somewhere. I mean, if you're if you're complete if your entire network is completely overloaded, for sure it's gonna take time, but then you may want to reconsider your network infrastructure

 

Carlton Gibson  31:17  

and spin up that extra node you talked about.

 

Markus Holtermann  31:20  

Yeah, though. So adding more nodes also means more communication.

 

Carlton Gibson  31:25  

Right. And you just hit the you said it was based on leucine. Which means Well, the thing that comes to mind is, well, Elastic Search, how does it compare to Elastic Search? Because I know a lot of people use that for high ingest and use case Yes,

 

Markus Holtermann  31:37  

in the very early stages of credibility. It was merely a plugin on top of Elasticsearch to provide a secret interface, which over time became a as well not a fork of elastic but I had a bunch of code from elastic. That was he used and when we quit, we had our own stuff around that are melted into the whole thing. And that made upgrade to be at that point. And then eventually last year, I guess I think we dropped in the core team they had dropped into dropped elastic. In that sense, that's

 

it's not in the codebase anymore.

 

So it's, it's not involved into direct development of the database itself. So it can't actually tell you that much about the details of the development and the specifics around to code.

 

Carlton Gibson  32:40  

But are you aware the sort of general like, what was it that elastic didn't scale in a particular way? Or do you know, to know that maybe you don't I don't know.

 

Markus Holtermann  32:49  

It's just interesting. I'm not entirely sure to be honest. Okay. Fair enough. Fair enough.

 

Will Vincent  32:56  

Well, yeah, and you I think your your master's studies you you did Some work on Elastic Search, right? So that's back in 2014 along with Alex.

 

Markus Holtermann  33:07  

And no,

 

Will Vincent  33:10  

no. Oh, whoops, sorry. I got the wrong I was looking up. I pulled up a weird. I'm sorry. I was just quickly pulling up. Wait, it's on your I am on my Masters without Alexander Greenberg, because I saw you how to talk on Elasticsearch and I was just trying to pull it up and I pulled up I did get from 2014 you have a whole post on it. So, why so you

 

Markus Holtermann  33:34  

know, so, you give a very good talking.

 

Carlton Gibson  33:37  

Elasticsearch,

 

Markus Holtermann  33:38  

it was just one cause at you need wavy. So, so almost almost always with all the projects that I had to do, I always try to use yet another technology just to like play around with it. And went from a C c++, Java, eventually Python two And then in the Masters net calls did this one on blog post it I guess you're referring to is the school data in Berlin. He figures Hey, how about we just build a static website and then so all the data they have into elastic and just create that straight from the front end. But that's about it. What I did for elastic now, thing I think you can refer to was a talk I gave on Heidelberg in Heidelberg. Tyler black. Yes, he probably likes to say no, what was it? And let me summarize what I remember. It was,

 

Will Vincent  34:38  

for me, I've got it here. It's combining Django and Elastic Search. Yes, it

 

Carlton Gibson  34:42  

is going to talk about how you can use the Django ORM and write that and then how you would sync to Elastic Search to your information retrieval, your search, yes. And I remember it being so good because I at the time I was flirting with this idea that I will just drop the RM load if we're going to have synchronous To Elastic Search and search from Elastic Search, and even we're going to serve the web pages perhaps from Elastic Search, why not just write to Elastic Search and not bother with the Django ORM at all? And you were like, No, no, no, because you lose the transaction handling. You lose the RAM, you lose all these. Exactly right. And I was like, yeah, that answered that question. canonically for me,

 

Markus Holtermann  35:19  

and yeah, the talk, I looked at the targets on the lookout for your data from Django con Europe in Heidelberg. You're right. There's 2018

 

Will Vincent  35:30  

but there's a great video functionless watch that on YouTube. Yeah, link in the show. What's the other talk project that I wanted to that I was I saw of yours recently was the the logging article where you, you know, you talk through Django and and logging at a really deep level, but then you just casually link to this get lab repo. That's amazing, where you have all these examples of how to structure your logs in different applications, which we'll we'll link to Like the, you know, cuz I put up code open source sometimes and, you know, you got a lot of good stuff in here, right? You have like an airline example a bank example. That's, and you just sort of casually mentioned at the end, like, you know, I would just like make a video of that being like check this work to make that

 

Markus Holtermann  36:18  

it's been a top of this this. So the topic there in this talk is about structured logging. And yeah, I could recite pause off the talk now, but then I'm not going to do that. I suppose you can get the link to the talk somewhere in the description. And I underlings

 

I feel like gonna spoil the surprise of this talk, but

 

Will Vincent  36:40  

Okay, well, I just I actually have code to back it up, which was, which I always like, like, it's not just talking. It's like you gave examples. Yeah.

 

Markus Holtermann  36:48  

Yes, the parts you mentioned that with airline and so on. It's the reason for that is because of the title and the cost. story of how I try to bring this whole title of the talk into why you would want to do structured logging. And what that refers to a movie.

 

Will Vincent  37:09  

Okay, well, we'll put the link up for everyone, so we won't ruin it. Or maybe we could talk about Django 3.1. The two things I'm excited about

 

Carlton Gibson  37:19  

Jason Pugh. So okay. 3.0 had the ASCII handler, which enables you to embed Django in a

 

ASCII

 

environment. So that's the foundation step. But 3.1 is the state where we can actually write an async def view. And okay, that's going to be basic, and that's going to that's gonna be fledgling as what we'll be able to do in future. But it's like, yes, so now if I've got my Django project, I need to make a few API calls out, and they need they could be done in parallel. Without even running a different service still would Gunawan still in the whiskey environment. I can write a parallelized view which fetches those rather than having to do those sequentially and that will be much quicker. And that's just a quick gain without having to do anything. Else. So, you know, this is the real start for Yeah, we can start to do these extra exciting things and Okay, will they there aren't going to be the async class based views yet that will probably get in a few, you know, when we all learn how to write these views, and we will learn what the patterns are. Anger support will evolve. But this is like the, for me, this is like, Yeah, actually, this is the releasing bit. This is the bit where we'd like Yeah,

 

Markus Holtermann  38:22  

actually, most of what you can do, and then and then actually possible and then and then after next versions, we need this three points, probably asking for database level, I guess, because that's still not right. I mean,

 

Carlton Gibson  38:38  

yeah, I mean, so yeah, this is the limitation is if you still using the the RM you that still needs to be wrapped in a sink or a sink wrapper and executed in a thread. So a lot of the things you want to do, they would still be sink. Yeah. But I mean, Andrew might Andrew Godwin wrote the views, PR. we merged it and he's quickly joking. All right now the ORM it's

 

Will Vincent  39:05  

Right, exactly.

 

Carlton Gibson  39:08  

It's gonna it's gonna be exciting. I think. I mean, this I'm excited. I've been right. For years, I've had to spin out a Node JS on the site before, before Django had days before Python had it set up to spring, spin up a Java JavaScript thing to do something async. And then pythons had async for a little while with async. io. And you know, there are other options, but async IO being the standard library option. And you think, Okay, well, I can just do it with async. io. Fine, brilliant. And there are a couple of web frameworks around that. That we've talked about Tom Christie style, it

 

is a go to that.

 

But to not even have to change your web framework to do the basic thing. That's Yeah, almost closed all the time.

 

Will Vincent  39:46  

That's pretty cool. Absolutely. And I think that because flask is working on this, but it's sort of like totally different things too. So not that it's a competition but I hope to get I guess, David Lord or someone on from class to talk about how they're Doing it, but it's gonna be another reason that Django just marches into the future. And, you know, when people say async Yeah, you can. You can do that, though, of course, like me and Andrews talks. To me, the interesting thing is, the use cases for async aren't as ubiquitous as maybe we thought they would, or the committee thought they would be five years ago. Yeah. Like that, to me is sort of the interesting piece. I mean, you know, in the same way that we have HTTP two, but it's not, aside from gaming and chat, maybe, perhaps, and, you know, in in IoT cases, it's being used, but it's not used everywhere. The The,

 

Markus Holtermann  40:37  

the interesting part about other part about asing is that it's really giving you a lot of benefits and when you have a lot of IO, and I see most websites, and huh, yeah, should chats and all these things. They are not particularly nice. Most websites in that sense, because there's a whole part of this media pot in there. But regular websites, it's, well, here's my my view that fetches a few things from the database and returns a bunch of HTML for the most part, or returns an API a bunch of stuff in a JSON response. As an as an API. There is not a lot of not really a lot of IO happening when you compare to a bunch of other scenarios. And yeah, you may get a bit of speed up here and there. But the occasionally, I think that the mental overhead that you need to apply there in order to wrap your head around a sink can easily reduce the maintainability of the of that page that

 

comes with pros and cons is pretty much everything.

 

Carlton Gibson  41:57  

Yeah, in the PR there was a note, Andrew put, I'm sure I'm sure it's there. It may not. It must be it must be otherwise I wouldn't be sure. But there was a note saying something like, we advise you to keep using sync until you've profiled. And you know that there's a real Yeah, benefit here. Don't just jump onto the async bandwagon because you think it's cool. I mean, it's like, yeah, that's not a reason to build your product in it.

 

Markus Holtermann  42:18  

Yeah, absolutely. It's, it's definitely an interesting approach and interesting way of writing code. And I mean, the whole front end part and bap has approached it for a while now and has been doing as in for quite a while. But then in the front end, you have direct user interactions where you want to do bubbles here and show things they are

 

synchronously

 

which you don't really have in a bap so our back ends with, like API's for the most part, anyway.

 

Carlton Gibson  42:53  

Yeah, I mean, it's there's some was it live, call it live view. So there was a phoenix Project which is in elixir.

 

Will Vincent  43:01  

elixir. Yeah,

 

Carlton Gibson  43:03  

yeah, they had a, they've got a thing called Live View. And then somebody ported that to Django. There's a Django Live View project, which is really interesting. I think they I think they're using channels under the back. But what's nice about it is that the the front, say a form can do validation on the server side, because it's doing what looks like local client side JavaScript validation on the form, but it's actually pinging the server and getting the, the response there. And one of the issues with writing decent JavaScript is always been I've got to rewrite my validation logic. And it's always been a bit of a well,

 

Markus Holtermann  43:36  

that's the Batavia project from from the pi b project, which aims at providing Python in the browser through cross like, I was going to say cross compiling to JavaScript. And nothing just wasn't me. I'm not entirely sure, but I didn't do it. How'd you remember how they how they built the whole thing? But essentially can write Python code? And then they have a Python? Yeah, but they have essentially a Python runtime. So no compiler or something.

 

Or repple. It's just the interpreter

 

that you throw some Python bytecode. And the interpreted is written in JavaScript, so runs in the browser. And then you run this Python bytecode in the browser. Yeah, that's kind of cool.

 

Carlton Gibson  44:32  

So it says part of the beware project.

 

Will Vincent  44:34  

Yes. Yeah. I was gonna say yeah, Russell's. Yeah. I think he referred to that in his keynote at pi con last year, too. We're probably yes. That specific. Yeah. Well, maybe for me like a final question. So this is something I was hoping to do in person at Django con Europe, but since we're in a virtual situation. So sitting on the board, a lot of what we do is the janitorial side of Django, paying the bills, but one of the things I want do is be a little bit more aggressive around. If we had a grant fellowship, if we, you know, didn't have to rely on Kickstarter for major new features. What would be some features that you think would be good for Django to add assuming we had unlimited people in money? You know, are there a couple that come to mind between the two of you? I mean, authentication is a big one, right?

 

Markus Holtermann  45:21  

I get a bunch of offcuts user overhaul is called with it, make it a bit more modern and adjusted to few modern security or authentication processes, stuff like auth or two factor or in whichever way authentication. I think the thing that's been on the docket for ever is a new admin. This this

 

Will Vincent  45:55  

while right, it's just a million dollars, right? The cost.

 

Markus Holtermann  45:58  

Remember who, who brought this number? I have the feeling this was because Jacob Jacob at a Jenga under the hood, I think. Yeah, if if if we have a million dollar a million dollar interview folks, I think he mentioned 10 to 10 people or so that spent a year then this is achievable or something like that. Yeah.

 

Will Vincent  46:22  

I just it's I like

 

Carlton Gibson  46:24  

a one on one. This is the off I'd like to see two factor auth built in I'd like an easy solution for users to be able to you know, have one time passwords have you know, yeah, all the other bits and bobs that come.

 

Markus Holtermann  46:36  

I don't think that's been a common

 

Carlton Gibson  46:40  

No, that's that. That's my one cell battery that like there are other things that we could add like nice little features that would improve it in a number of ways. But my one big missing battery at this point is to FA have some kind of modern, you know, use your use your smartphone with its bio thing. Registering security may And know the issue that's logging in? Yeah, I think we could, we could do that. Now. There's I mean, it's a big job. But

 

Markus Holtermann  47:07  

it's it's another thing that's been

 

floating around and a fairly long time, I think kind of was dropped to a degree once the whole channels on string of channels Cameron, and absolutely once once it was clear that Django itself is going to have some amazing capabilities, is a API for task execution. So not the execution built into Django. But an API where you go and like, I want to put sent off this task and thing, and then whatever implementer implementation project of our project is out there that wants to implement that can use on convert on top of that. So your wallet, just to pick one salary gengo integration foot, leverage that task. That's what this task interface that Django cook anyway, couldn't leverage that task interface to Django users. And you define whatever this one setting in your in Django and on salary. And

 

again, quote unquote, just works

 

Carlton Gibson  48:25  

like a rubber but the idea is that you could replace celery with whatever and our queue or I don't even know what's what else is out there. We might have an SQL back end, you might have a radius back end, you might have a, you know,

 

Markus Holtermann  48:38  

well that's that might not even be the level of where the integration goes. But this is it's been floating around like five years ago or something. I remember having a discussion with that in Cardiff at the Django calm Um, so that that was 2015

 

Will Vincent  49:01  

Well, I hope that this is something that I hope to encourage more in the community because on the DSF, side and treasurer, if someone said, we need this new feature, raise X amount of money, there's a number of ways we can get that money. So I don't want that to be the limiting factor. I hope to encourage that, you know, we don't rely on kick starters. But it's sort of it's sort of the chicken and egg where I mean, because the total budget for the Django Software Foundation is around 200,000. Most of it goes to the fellows, then to conference workshop sponsorship, and that's a teeny tiny amount for the ops and, and whatever. And that's it, because we don't need more and we're not trying to raise a ton of money. But if some feature was $50,000, you know, whether it's getting a grant from Google, which is provided help or some of these corporations, or you know, a number of other ways we can

 

Carlton Gibson  49:54  

try to raise money.

 

Will Vincent  49:55  

We can raise money in a way that doesn't go against any of the principles of Django, but we We need to know that we need it. Is that the first step. So

 

Carlton Gibson  50:05  

that kind of brings us full circle because one of the things I'm really excited about the depth 10, which is the change to Django governance is that the technical board are meant to have a new role of helping to guide the technical direction of Jango into each major release to sort of have a take a bit of time, solicit ideas, compile those ideas into a bit more of a feature roadmap, because we've got this time based release, and we'll keep that but the technical board will hopefully be able to lay down some that we're aiming to improve country both were aiming to, you know, introduce a back end system if that's what the technical board, come to the conclusion without consulting with the community, then we will have that, that list of ideas which are that list of things we want, which then you know, as a board or the membership could act on.

 

Markus Holtermann  50:53  

Yes, absolutely.

 

But that also

 

requires like From outside, or from people using Django or from people in the community. So to go, like, hey, this would be cool. And then while we, I guess, for some obvious reasons, we can't do all the things. And we probably shouldn't do all the things. But like the the things that are useful for the majority of people, if we have, there's a possibility to build a API in Django or build the bots and bits and pieces into Django, that then other third party applications can use and leverage that to build a more sophisticated paths to be used as, as applications. I guess that's that's definitely something we have found. We definitely will rely on the need to rely on the community to bring up ideas there.

 

Will Vincent  51:50  

So one of the things I'd like to do this year is to do a Django survey again, which I think Tim Graham did in 2015, something like that. similarly to the Python survey, we're in addition asking people how they use Django, we could, we could have a place to say, you know, what would you like to see in part based on your use case at your company? Because we're sort of getting at is that a lot of the times we're flying blind in terms of Django usage? Because we don't track? Yeah. We don't have tracking in there. So we don't really know. A company's using Django unless there's something open source or someone from there tells us by design, but that does limit sometimes the new features and things that we can add.

 

Markus Holtermann  52:29  

Yes, absolutely. This.

 

Another was a discussion around integrating threat, some kind of usage striking in Django, which was, I think, fairly strictly. Let's call it shut down by the by the core team. I think there's a depth out there. Like that proposal, like

 

four years old. I'm getting Yeah,

 

Will Vincent  53:00  

yeah, I think it was it was shut down. I mean, because we can you can see on pi pi, you can see the number of Django downloads but the Pi Pi stats are there's many ways that those are unreliable actually. So I mean, I, I think our take to be conscious of security of privacy and all these things is the right one, but it does mean it takes a little more work to find out what, how people are using it. I mean, literally, I mean, Jeff Triplett and I are trying to Jeff triplets been working on a list of like the top 500 companies using Django. And it's a bit of work to find that in part, so we know who they are that we can ask them around features we could, you know, try to suggest sponsorships and stuff but like we have no idea who they are. Just Yeah, without doing the work. So the manual work. The

 

Carlton Gibson  53:47  

depth I was mentioning is DEP eight. Gathering Tango usage analytics is difficult. I mean, commercial companies obviously, often put this kind of analytics in their in their packages, you know, Download VS code, it's got analytics, which goes back to Microsoft. That's not necessarily a problem. You opt into all of that, and you know about it, but

 

Will Vincent  54:09  

one is gonna have is it like? I think that I mean, we could message I mean, for me, we could totally read someone could we could look through Django project in general, from scratch with someone with fresh eyes have a Doc's fellow or team. I think there's a lot of ways that we could improve that and talk about a lot of things that Django project and team members are doing that we don't spend the time talking about, rather than just doing

 

Carlton Gibson  54:35  

for this capacity as well. Like you, you know, that there's a group of contributors who are very active and there's new people coming in and, you know, some common stay some come for a bit and leave some come for a bit, leave, come back again. You know, everybody's working quite hard on keeping Django moving and it moves fast. You know, look at the last few releases. They've all been, you know, just major, great relief.

 

Will Vincent  54:58  

Yeah. And that's and I hope That I want to call out more attention to the people doing that work because I know that everyone who's on you know, our mailing list knows who one another is, but most people don't know. And I not that people do it for that reason, but I think it's important to, as to your talks, Carlton, right, we need new contributors. There's a lot of strength, but there's quite a bit of fragility when it's a half dozen people doing. Yeah, you know, two of you are on this call are doing a lot of the work

 

Carlton Gibson  55:27  

and saying, Oh, cool. It's a video call. So if a meteor strikes one location,

 

Will Vincent  55:35  

but anyways, but I mean, certainly I view my, you know, my role in the board as being a janitorial, a support role for the court, you know, the technical team and, and trying to enable that. So,

 

Carlton Gibson  55:46  

but yeah, I mean, we've So okay, let's, that's a good point to wrap up, because, uh, Marcus, thank you for your work on the ops team, on security team, on the technical board and everywhere around the Django guns and on the community as well.

 

Markus Holtermann  55:58  

Thanks Call for having me. Thanks, everybody for still contributing to Django and like, keeping this project running. It's a pretty impressive what's happened for the last

 

16 years now? 15 six. Yeah, something like give or take. Yeah. Yeah. Well, it's quite a while.

 

Carlton Gibson  56:25  

It's gonna be a few more yet won't things.

 

Will Vincent  56:26  

Yes. All right. Thanks again. All right, and everybody. We're at chat, Django, on Twitter, Django chat, calm and links to everything in the show notes.

 

Carlton Gibson  56:35  

Join us next time. Bye bye.