Calvin is the co-founder of Six Feet Up, a Python/Django consultancy. We discuss the upcoming Python Web Conference 2022, a recent lightning-prediction project, and current best practices around Django APIs in production.
This podcast does not have any ads or sponsors. To support the show, please consider purchasing a book, signing up for Button, or reading the Django News newsletter.
Will Vincent 0:06
Hi, welcome to Django Chat, a podcast on the Django web framework. I'm Will Vincent joined by Carlton Gibson. Hello, Carlton.
Carlton Gibson 0:11
Hello, Will.
Will Vincent 0:12
This episode we're very pleased to welcome Calvin Hendryx-Parker from Six Feet Up. Hi, Calvin.
Calvin Hendryx-Parker 0:16
Hey, Will. Hey, Carlton.
Will Vincent 0:19
So it's been over a year since we last spoke last time we talked about has conference, loud swarm projects you're working on. The new version of Python web conference is coming up very soon. So I thought I'd ask you say a little bit about that virtual conference and what people can expect.
Calvin Hendryx-Parker 0:34
Okay, this is the fourth year yeah, fourth year for the Python web conference. We started in 2019. Pre pandemic, but fully virtual. It's always been a virtual conference since the very beginning, we felt it filled a gap, where there weren't a lot of web talks that seemed like going on at various other conferences. And given opportunity for those who couldn't normally make it to in person conferences, a great place to kind of gather and see great talks from the people in the Python community who rock doing all kinds of work. So this year, is going to be March 21, to the 25th. So if you're hearing this, now, go grab a ticket really quick, there's probably plenty of space left. Since it's a virtual conference, we don't have constraints like catering and things like that, which is nice. But we will have definitely more attendees this year than last, we've already had about 3x, the signups that we have of last year, which last year was double. So I was super excited about the growth of the conference itself. The speakers this year are fantastic. If you if you go to look at Python, web conf.com, you can see on the speaker's page, it's my favorite page to go to, because it is a beautiful page of just amazing folks who have who've, you know, given their time to come and speak at the conference. And I think you'll find it's just not your same boring old panel of people. There are really diverse, interesting folks from all backgrounds telling all kinds of stories, we're gonna have, you know, two tracks of APT of we've got a cloud track, a PI data track, and a culture track. And then we added in the tutorial track, two years back, and so that's actually running all five days. So we've got six tracks. Almost all day, it's a kind of starts in the morning, Eastern Time goes till about one or two in the afternoon. US Eastern trying to make it friendly for most folks who could join boy expecting people from all over the world we've had, you know, 30 plus countries last year, you know, almost 20 timezones covered for folks who are joining. One cool thing with the platform run, it's built in Django, the loud swarm platform actually allows for the recordings to be posted right away. So we're actually you if you miss a talk, or there's conflicting talks, in about 10 minutes after that talk, those talks are online. And actually, people are time shifting and watching, you know, 24 hours a day, and they're all hanging out in the in the chat, we just use Slack for chat. And so you'll see, you'll wake up in the morning and find people from all over the globe, chatting about the talk, they are watching or or Python or open source or who knows what, we've got some really cool socials planned as well. So each day, there'll be some kind of a social activity. One of our speakers is giving a talk on burnout. And she's going to do a breathing exercise and a mindfulness exercise that I'm really looking forward to. I think that's really important for folks during these obviously troubling trying times, for us to focus on. If you can't focus on yourself and help yourself first, you can't help anybody else. So you need to make sure that you're taking care of yourself. So we've got a whole session on that, plus a social dedicated to some mental health. I'm looking forward to that. The keynote speakers, again, amazing group people. The last one on the last day is a gentleman I met out in San Diego in the last year, and he's going to talk about a project. He's been working on a real passion project for him. And I think it should hopefully, I think it'll bring the room to tears, the virtual room to tears. But yeah, it should be tons of fun. I'm super excited. The community really shows up for this one. We've got over 90 speakers, which is I mean, I think last year we had 60. So we're looking at, you know, 50%, more, more everything. It's all all more.
Carlton Gibson 4:12
Nice. Fantastic.
Will Vincent 4:14
Yeah. That's amazing conference. It's it's been interesting, too, because everything. Everything's been virtualized, a couple years, the Django cons have been, I think both US and Europe are planning to be in person this fall. But it's, you know, even hopefully, once we're back to normal, there's something to be said for a virtual conference, because you can have more speakers doesn't limit people. I mean, it'd be interesting to see if there's some hybrid approach or how that how that works going forward. Because a lot of the all the Django cons they they post the videos pretty quickly thereafter to YouTube, but there's something about Yeah,
Calvin Hendryx-Parker 4:46
I'm the media person, but in two months into April, yep. So I'll be there. Yes. But Python Python conference will always be virtual. That is the format that we want it to be in. We really want to reach Folks who couldn't normally come in if you have a need or some means by which you can't make it to the conference because of financial constraints, let us know. Reach out, the group is definitely willing to make sure that all the people who want to be in the room can be in the room. And I'll be a Django con as well. So I'm looking forward to hopefully seeing you all at Django con us.
Carlton Gibson 5:22
In Portugal, are you just San Diego?
Calvin Hendryx-Parker 5:26
Oh my gosh, that's that's definitely my bucket list. When they announced I was gonna be in Porto two years ago, I got so excited to go planning my
Carlton Gibson 5:34
trip. Today, it was first announced.
Will Vincent 5:38
You're gonna do a whole caravan Carlton, right. Take all your kids. And
Carlton Gibson 5:42
yeah, we'll have whether that happens because they're back at school. So I might have to go back and say, What? What a bummer. Yeah, dang it. Oh, God, I wanted to ask you, Calvin about the last one platforms. We did talk about it a little bit a little bit. The last time you're on the show, but I'm sure it's come on in the last year. And you've you've made enhancements to patching talk technically there. But perhaps as a kickoff, I wanted to ask about what was the motivation for loud swarm, creating Python web conference was that,
Calvin Hendryx-Parker 6:13
oh, he was 100%, the need for a better way to engage during virtual conferences, the bulk of the platforms are pretty rigid, and don't allow for a lot of that kind of tightly coupled engagement between chat, the speaker q&a, the podium experience afterwards of talking to people, you know, the whole social aspect of it. And we saw, there's definitely had to be a better way to do that. And so we built one, it's been really interesting journey. The technology we used underneath the covers for allowed swarm actually has helped us build other projects for other clients, which I'll talk about today, because I think we're going to talk about one of our other projects that we worked on. And a lot of the ideas that got spawned during the loud swarm project, have now shown up in a lot of our other projects, especially when it comes to things like serverless, and scaling Django and doing things like WebSockets. And, you know, real time, those are all really interesting topics came out of that. But yeah, the, the loud swarm experiment has been totally fun. It gives our developers a chance to kind of, you know, breathe a little bit and stretch and, and work on technology. That's, that's a little more interesting than some of the other client projects. Not that any of our client projects aren't interesting. But there are definitely aspects where they would love to try things, and not have the risk of it being on a production project for our client. But okay, something that's really owned by us and built by us.
Carlton Gibson 7:34
I think that's a really interesting thing for just from a developer point of view is because you know, Django excels at solving solving the web problem quickly, like you got a template view, you've got a template, you get your HTML out the title in some JavaScript that static files does its things problem solved. Oh, well, what did I need? I need a template view in a URL come? Well, that wasn't that exciting. But then, you know, there's all this other stuff you can do, which you hit basically a whole bunch of buzzwords there. I don't mind which of those should kick off.
Will Vincent 8:00
Well, Carlton, you're maintaining channels, currently, I believe, right, are helping to?
Carlton Gibson 8:05
Well, I'm sort of not maintaining channels at the moment. Yeah. Keeping channels? Yeah, I'm keeping channels alive. Basically, I, it's very much like, Okay, this is where I want to spend my time. This is where I want to, you know, my my volunteer, open source, but there hasn't been any of that time for probably six months now. And okay, that will come back. And, you know, I take it over. But yeah, channels is in a sort of maintenance mode at the moment, like, close a few issues and make sure it's compatible with the latest version, but it's still waiting for,
Calvin Hendryx-Parker 8:36
you know, the next No, I feel I feel you on that one i I wish I had more time to spend on it as well. I feel like channels is a almost like critical technology for Django, due to the real time WebSockets and the fact that applications are more more like applications than they are stateless web web pages and
Will Vincent 8:55
are using channels or what are you using for the web sockets on load swarm?
Calvin Hendryx-Parker 8:59
Oh, yeah, absolutely. 100% channels, we actually just so the latest rev of the stuff we did was mostly to bring everything up to date. So latest versions, Django latest version and channels, latest versions of flower, which is a troublesome story, you know, because those things, you know, the various Carlton laughs because he knows the version dependencies between these things are sometimes tricky to get to get upgraded. Right. So most of the work has been done on just shoring up the platform making sure it's always on the latest stable updates of things. Same thing for the front end, so it's a React front end. So we updated all the React bits here and there salt some you know little niggly type bug things but generally the platform is is how I want it like it really solves the problem well gives people a great place to come and view content together and in chat and socialize
Carlton Gibson 9:48
to cannot ask a question as a consumer of channels right? So you loud Swamis consuming channels and I look at it and I think you know, basically it's kind of okay you know, the core consumer layer that the ideas They're nice and solid. And there's not much to it. And it's kind of simple. And what's missing is like the the maturity and the the extra features around the top. Would you think, would you say that's fair? Or would you would you point to No, actually, this is where our problems are. And we need that fixed.
Calvin Hendryx-Parker 10:15
I think that's 100%. Fair, I think the simplicity of the framework allows it to be used in any way we want, much more easily than if you had kind of been a little more opinionated with certain features. So the fact that a consumer is just dead simple to write, it makes WebSockets much more approachable to folks. So if you've ever tried just talking to a raw WebSocket, with like, you know, WebSocket, cat or whatever, on a command line, and you just get a stream of junk back at you, it makes all that a lot more approachable. So some some open source projects may be feature complete. And as they should be, I don't know if we've got a bunch of wishes for it unnecessarily other than to keep it easy, simple. It's easier to understand these are then probably maintain, easy to keep keep it secure. I go okay, I got one I got one for you is probably and he gets the issue really doesn't it's not channels issue. It's it's WebSocket issues is that there's no formalized, like authentication mechanism. So a lot of times you're contriving your own means of validating or authenticating or authorizing who should be connecting to a specific channel and what what kinds of things they're allowed. So you have to basically come up with your own protocol for that. Maybe some opinions are needed there to just guide new new folks in that realm. Because it's not straightforward.
Carlton Gibson 11:32
No, I mean, I don't have an instant story as well. I mean, a lot. There's a lot of the issues that come up on the repo things like getting it working with nginx, or get, or a client keeps disconnecting. And you know, what JavaScript libraries you're using? I quite often I'm a bit like, I have no idea what maybe you should use? I'm sorry, I do. I just don't know.
Will Vincent 11:53
Calvin, can I ask about how the deployment so you have the Django back end toasted on server, and then the React front end? What are you using for the React front end for hosting, because there's, you know, there's Netlify, there's all sorts of options. I'm just curious, your setup.
Calvin Hendryx-Parker 12:08
So he in the end, there are no servers involved for the lightsworn project. And for this, like other project I wanted to talk about, which is actually quite beautiful. I love the fact that we don't have to maintain Ubuntu machines and go in and patch them. We do you have to rebuild images. So it's all done with Docker images that are deployed onto fargate. in Amazon, which is basically a more easy version of Kubernetes. That's probably doing it a disservice calling it that. But you I want a place to run containers, I want a place to run my celery tasks, I want a place to run my channels, workers. And fargate makes it very easy to deploy the backend pieces like Django, and the workers and celery and all those pieces just as tasks as what they're called inside of the fargate ecosystem. And for the front end, we're just using React that's hosted on s3 bucket. There's nothing fancy about the front end pieces, because they all just it runs solely in the browser. It connects back with WebSockets through CloudFront, and the load balancer, and we actually use the load bounce we split the load balancer in to so that WebSockets traffic goes to a whole separate pool of workers for the channels bits. And the API calls all go to a dedicated set of workers just for the Django bits. They're using the same image. But they have different characteristics. And they need to scale differently based on memory usage, especially in the case of WebSockets. Because each socket connection takes up so much memory, you want to build scale that separately from the API servers, which actually are pretty low utilization. Most things are cached when possible. We use a lot of redis to cache complex queries back so that you know anything coming in for like the big schedule for an event comes most of the time right out of Redis.
Carlton Gibson 13:56
And can I just saw set up using whiskey for your base API, and then ASCII for the just for the async website. WebSocket connections, it's all ASCII.
Calvin Hendryx-Parker 14:06
So it's all Daphne, also using Daphne on both both WebSockets and for the Jango. Front and other we're not doing anything fancy on the Django side, as far as async, other than talking to celery workers and the the channels workers. So I think we talked last time, we were doing a discord integration on the back end for last form. That celery work that's not a worker that's actually a channels worker. So we're using the background workers feature of channels, which is pretty awesome to basically start things on run. So when Django starts up, there's no easy way to kick off a request to something or maintain a, a stateful connection to anything other than like a database connection. This allows us to actually say on Run, run through these set of asynchronous tasks, and some of them can be short lived and some of them could be long running in That's how we start up and actually connect to discord servers to establish a web connection, a WebSocket connection that is server to server and not server to client. Yeah, well,
Carlton Gibson 15:09
I'd really love it if you could write, you know, a company blog or something, write something up about this the architecture here, because I think within the ecosystem, there's real lack of, like patterns. And you know, everybody's still breaking. Yeah, the pottery, the snow kind of thing
Calvin Hendryx-Parker 15:22
is that is true, I guess that's basically one of the things that being opinionated versus being, you know, a little more open about how you do things for. And it's unfortunate for new people, it's hard to grasp that. It's hard to not know, like, what's the what's the one way? There's not always Oh, way, but there's definitely some patterns. So I think that would be a good idea.
Will Vincent 15:41
That rings true. I just, I just did the update to my Django for API's book. And, you know, the only way I know is just by asking around to people, you know, it's much simpler. But still, the new update gets more complex. And it's like, people have questions. And like, you know, if there's if there is a consensus in the community, I'll tell you, but sometimes there isn't. Right. So maybe as a lead in speaking of case studies, you do have a whole bunch of projects on your site we'll link to and one of the ones we want to talk about, I don't know if this is switching over too early. But the flash predicting lightning strikes,
Calvin Hendryx-Parker 16:12
yes, this is such a cool project. So for folks who aren't aware of six feet up, you know, we're just a Python just we're a Python cloud consulting company. But we recently, last couple years, have set like our tenure objective, like what is what is 6.1? I accomplished, of course, across a 10 year timeframe. And really, it was to get 10 impactful projects in 10 years. So when we think about impactful projects, we are trying to define a do gooder. Like, what is it? What does it do good are these days that it took us two years to try and define that. And we still failed at defining do gooder. But we did define what impactful meant us. And so the project I want to talk about if you go to our site, there's a there's a link for our impactful projects. It's basically an acronym for criteria we use to measure a specific project against to see if it's impactful or not. And the one that you're asking about is really cool, because it's doing something that you know, as a kid, you're obviously fascinated with the weather, and or even growing up like you see lightning and thunder, although maybe most people don't most people do or don't I grew up in the Midwest. And so there's that, yeah, there's lots of awesome thunderstorms. And so you'd watch these things roll by as a kid, and I just obviously, infatuated with it a little bit. But we had a friend of ours from the Python community call us up and say, Hey, I've got a friend who wants to basically take a Python notebook where he's developed an algorithm for predicting lightning. And you may think, well, haven't we done this. And we haven't, all the lightning watches and warnings that you get, are based on knowing where lightning is striking now, not where lightning will be. And so this is actually a new thing that doesn't have that existed until we helped him put it into a production state. So he developed this algorithm in a I just a Python notebook, a Jupyter. Notebook. And obviously, you can't deploy Jupyter Notebook into the cloud and say, start start selling this as a service to folks. But he has come up with a way to give basically 99.6 accuracy and a 15 to 25 minute, lead time, like so you can say within you know, 99 plus percent accuracy, lightning will be here. And within a sock camera with a radius is but just says yeah, just it's pretty accurate. It was kind of interesting, you know, as we're working through the project with the client. At some point, he was actually with a race team down in Daytona for the Daytona 500. They had him basically in the pit, trying to helping to predict if you could, if you're not familiar, Florida, Florida is one of the lightning capitals of the world. And so they run car races in a lightning capital the world they want to know when this is coming. So they know how long they can go until they like kind of call things. And that's actually one of the big reasons for this project is that there are lots of activities where if they didn't have to if they could shorten the window by which they had closed down certain kinds of activities, like planes taking off or landing, or sporting events or construction equipment, being energized or not energized because there might be lightning in the area. I mean, the amount of savings globally, it's kind of ridiculous. Like it's a very, very large number. I hesitate to even fathom, I guess, at what that would be. But the ability to not only just save lives, like if you know little league games are going on and you can accurately say we probably should get everybody take some cover, you know, 20 minutes early to the fact that you can save money from the delays and flights cause millions, hundreds of millions, if not billions of dollars to the economy each year. And a lot of times they can't they have to be conservative. Obviously there's the people in planes Did it it's, it's a big deal. But if you can narrow those windows, the the amount of money and time saved by people, it was tremendous. So that's why this project was important it because it has this impact on the planet, the globe, people, there are now, you know, there's obviously more commercial sides of this, where if I'm protecting if I'm an insurance company protecting giant equipment on construction sites, if that equipment can be de energized, when lightning hits it, then there's no damage or less damage to a piece of equipment, apparently. And so they want to be able to use that information in an automated fashion to automatically shut down or turn on these DC pieces of equipment or tell the airlines when they can and can't open up their windows for people to take off.
Carlton Gibson 20:46
Fantastic. Also, that'd be kind of handy Back to the Future type scenarios where you know, things out your DeLorean.
Calvin Hendryx-Parker 20:54
It's kind of cool, because when you're working on the project, you start using the API yourself to be like, Hey, I wonder, you know, he may hear a little rumbling, you know, in the in the sky. And so we will go and run it and see like, where the lightning is coming? And when is it gonna hit?
Will Vincent 21:08
That's, that's such a cool instance of using, you know, Django to make a web version of something else, right, like I'm in, I'm in Boston, and there's all sorts of science and medical stuff. And I meet people all the time, who are, who want to put it up on the web. And you know, there's less of a need of real time or these other things, but even just taking a Jupyter Notebook and putting it up on the web, you know, is is challenging. And there isn't a one size fits all solution, though, that's for sure. But that, you know, every, every stem grad student or researcher wants, you know, a web version of what they're working on. And if someone could solve that drag and drop, that would be worth a little bit.
Calvin Hendryx-Parker 21:45
I hope. There's no one size fits all. I think a lot of times flask is probably an easy answer for these kinds of things. But maybe you don't have authentication needs or maybe your needs aren't sophisticated, you just need to get an API on the web, where choosing like Django was is an easy choice. Django rest framework is super easy to get, you know, something off the ground with quickly. We're using the KNOX API tools. Yeah, we're doing security. So you can actually easily rotate API keys and like enforce single clients at a time authenticating and using the API a once and also kick off people who shouldn't be on there. So it's a nice tool for generating things like API keys, because API keys when you think about doing API's, the funnel different for me, the first thing doesn't come my head is how do I'm going to authenticate and authorize I authorize people. I just want to like build the cool tool that's going to return some cool data to folks to do something interesting with now I got to figure out how to like, Well, how do I like, track their usage, maybe build them how to like, protect certain usage levels, all kinds of like that back office stuff that goes into making a thing real.
Will Vincent 22:57
So the only reason I know of Django rest Knox, I had a version of it in an earlier version of the API's book. And I think actually, maybe I refer to it when I did a Django CON talk on auth. But that is the Yeah, the challenge of, you know, JW T's that's one of the Appeals is you can time limit them and stuff, though. There's I actually, I'm curious to ask you, it's almost like the Docker question, how do you feel about JW T's? Because I feel like everyone is on board with them. And my sense is now people are like, a little less completely on the JDBC bandwagon. for authentication.
Calvin Hendryx-Parker 23:26
It feels like it's all over the board. We because, you know, obviously I consume API's and various projects in it seems like every org has varying levels of sophistication with doing like API tokens summit still, like basic auth. Like how man, like, I can't believe we're still here, the bearer tokens or doing like the, you know, the rotating bearer tokens, where that seems to be a good pattern, that it is easy for folks to like, comprehend and use, like, it's easy to plug into postman to try it out. Like you're not, it's not a stretch. I don't, I don't have a huge opinion on it. Other than make it easy. I mean, there's some some systems out there where I've had to like, sign the requests going back with the cookies and the tokens, and then you get back a signed response that you have to then validate the signature. And it was all for like a video streaming service with nothing sensitive. Like I was like this, this is that extreme? for that? You know? It's hard,
Will Vincent 24:27
though, I think, right? It's just like, it's just a total like, Oh, my God just make it work. Even for otherwise knowledgeable clients, I guess,
Calvin Hendryx-Parker 24:35
right. I mean, for loud swarm, we kept it simple. And same thing we did for this flash project, which was the Django rest Knox tokens. Let Knox manage the tokens. So it has a API endpoint for authenticating. And then once you authenticate, you get your token. And that point, you can manage those tokens in the parameters of those tokens with with Knox, which is I found to be really just a nice, elegant solution to that problem. Now on the front end, we also used I think, was Django sesame, for doing like one time passwords. That's
Will Vincent 25:08
Aymeric project, I
Calvin Hendryx-Parker 25:10
think, right? Yeah. And that's, that's pretty cool. We had done a project in the past where we tried to go password lists, you know, where you basically put in your email address, get the token over and your email, come back over and like you're logged in. Email is still such a mess is just, I wish that would work. But it really was a headache. Now, we still use it for things like password resets, or invitations, that's where we use sesame. So it's really easy to send out a couple 1000 invites to people for an event, they click on it, it's already got them logged in, they fill out their profile, they click Accept, and they're in, I get that that user experience is so smooth, as opposed to like, a lot of the signup experiences, you may get a different site. So using the combination of sesame for getting the invites out and the knocks for the front end having the token. We don't use any cookies in Lodz form application at all. It's all using local storage with KNOX tokens. So maybe we're a privacy first event platform, but there's no no tracking cookies. I mean, there's tracking obviously, going on when you watch a video, so we can know how many people are watching, you know, what videos and how long they watched for and things like that. But that's all done mostly server side or with the player?
Carlton Gibson 26:25
And it's sort of anonymous, right? It's not like oh, yeah, it's It's Julie that's watching this video. It's just a number of users. Yeah,
Calvin Hendryx-Parker 26:31
actually is anonymous, I don't believe we tie a specific user to what their viewing was, we just want to know how many people were viewing a specific track or specific video.
Will Vincent 26:41
So you mentioned postman, is that your preferred client for consuming API's? Because there's, I just saw in the orange site, there's a new open source one, there is power for Mac's, which are just curious, and as a practitioner, what do you like,
Calvin Hendryx-Parker 26:56
I do like postman. Also, I guess, I've kind of split 5050, I'll use postman for this real quick, I need to like launch something. But if I'm actually actively developing on a project, I'll be using a PI charm user, I like pi charm, pro a lot in the HTTP client built into pi charm is pretty awesome. Because you can actually have it run through and do full, like, save sessions to English and postman too, you can save off variables in postman, and then like, reuse them later in subsequent calls, you can do the same thing in PI charm, but just as plain text. And so instead of having to kind of drill through gooeys, to set everything up, you can actually set up variables and, you know, do the login, get the token, then reuse the token to like now call these you know, 10 calls afterwards and simulate a real set of transactions back and forth between you and the server. And you can actually do assert assertions. So you can actually make kind of unit tests like bits in PI charm to assert, and actually, postman has this too, they both got similar sets of features. But one's more kind of graphical, if you're, if you're not, if you kind of intimidated by lots of text, maybe you're new to it, or you just want a quick and simple way to do it, postman's way to go. If you're really diving in deep and really want to, like, you know, check this stuff into like source control and have it right alongside your project. The PI charm HTTP client is really nice.
Will Vincent 28:19
So it's called hopscotch to PS. That's the new one. Oh, fancy. Wow. I well, it has 38,000 stars on GitHub. So I don't know how new it is. But it's an open, open source API development ecosystem. So I guess it's not that new. But,
Calvin Hendryx-Parker 28:34
I mean, it looks cool. Using HTTP from the command line to Yeah, quickly, you know, ping against something.
Will Vincent 28:40
Right. Right. I mean, it depends the tool for the job, right? Yeah. For sure. What about you, Carlton, you're you're building but Biden, you know, you API's
Carlton Gibson 28:51
I use, I use, I've been using pour for, which is the Mac native one for years and years, it's the same kind of thing, it's got a request builder, and you can customize absolutely every aspect of the request. And you can view the request the response in different ways, and you can, you know, extract a bit and then it will want one thing I really like is at the bottom, it's just got, you know, a drop down for expressing this in in whatever so go, or requests or HTTPS, or Swift or rust or, you know, and, and I find that really handy because, okay, you're sort of prototyping in poor. And then, you know, cut and paste its into into your project as working code. But yeah, they all of these, all of these tools have very similar capacities. But that kind of HTTP laboratory environment, you kind of really need when you're developing an API because the docs are never quite as good as you want them to be, you know, even like, take stripe, they've got the best API. Now. They're not they're kind of great, but you still have to make the request. See what comes back work out exactly. What part it is you can't just go from the API documentation. So yeah, I use port. But I'd have been doing for a very long time. But yeah, they're all great. I'm gonna go try the Python one. Next. I didn't realize it was built in the so many things. Yeah. Whenever you're always discovering,
Calvin Hendryx-Parker 30:17
if you go that like the dishes in the new menu, like new HttpClient, and like eat a blank editor, and it gives you some samples, or you can paste in a curl, and it'll convert it right into the code for you. It's kind of cool.
Carlton Gibson 30:28
Okay, I'm gonna give that a try. That sounds good. So can I ask the flash? You said the flash. The tech on Flash was built from what you've been working on allowed swarm. So is that built around channels and as well? Or is it more the serverless deployment and it's more
Calvin Hendryx-Parker 30:45
Yeah, more the serverless deployment aspect, that specific application, I don't think it uses WebSockets. Yet, it's mostly an API consumed by their customers. So they sell it as an API and to give people access keys. And there's minimal management, there's no real fancy front end application around the front of it's not a consumer app, it's really a business to business type application. The pieces we leveraged from Wildstorm, though, are definitely the fargate deployment technologies and techniques. This project goes a little step further in that we're also using lambdas. I don't think lots Blandford lots from does use a couple lambdas here and there. But it's not in the main flow of the application where this one actually is in the main flow. We have lambdas listening to SNS topics, which are just basically messaging coming from amazon for weather data. So the National Weather Service, the NOAA has a bucket of public bucket full of weather data, which is really again, so cool, you start exploring some of these areas that there's like this open data, I mean, just like terabytes upon petabytes upon petabytes of weather data are available to us as a human being on this planet. It's all open and free and accessible via API's, or just it's just sitting in an s3 bucket. So we we take those notifications, there's a real time feed and an archive feed, I say archive and air quotes, because it's like every 10 minutes is the archive feed and the real radio. Yeah, it really any real time fee is a few times a minute, you get the radar sweeps that are that are coming through. So we actually currently go off the, I believe the archive data, because if you go the whole technology problems behind the real time part of it. But with that archive data, we now have a lambda that can go grab the data out of the bucket, pre process, do some some level pre processing against it, throws the results into a Redis. And then that's available immediately for the Django API now to consume and reuse and republished back out. So a lot of these things, it's about how can we speed up that prediction process, when we were first handed that notebook to take archive weather file and process it into the model needed to then do predictions against, we were talking in the order of like minutes, I can't remember maybe in the case study, we said how many minutes, but it was, it was not a fast process for them to process it, we got that down to in seconds. Now, using lambdas. Below scale that out changing up some of the, you know algorithm to be a little more efficient. It's a great combination, having like scientists working with real software developers, because scientists have amazing ideas. And when they can explain it to a developer who can truly take that idea and accelerate it on the web, their eyes just light up like crazy, like, I can't believe this can actually happen so fast, or we can deploy it to, you know, into production and be able to use it. So that's really fun. Like, I love that part of the experience of
Carlton Gibson 33:41
software developers dream, right is that, you know, you read all these books on software development site, well, you need to talk to the domain expert, you need to kind of model the domain and you need to, you know, create the, the programming structures that map to the domain, and you never do all this. And then you do do it. And it's like, yeah, this is why
Calvin Hendryx-Parker 34:00
that's how it works. It's beautiful. And then then wrapping all those trimmings around it. So we use a lot of the cloud native tools in Amazon for this project. Whether we're deploying the the main library, we actually extracted the part of the notebook that was like the important bits into its own dedicated Python library that we deploy into code artifact that is now used in the building of the couple containers for us. So we have a container that runs the Django bits, which is the API. And then we actually deploy a container for the lambdas. A year and a half ago, and Amazon announced support for Docker images as a runtime for lambda, which is we have a really good blog post up on the 16th site. Because it's not straightforward. If you've got C dependencies. This starts to get a little tricky, so packaging it up in a way that actually works. If you just got a simple Hello World, you know Python, you just use the base Python image for lambda and on your on your way. But as soon as you got something where you need to do some See compiling. And like, for example, using NumPy, and pandas and things like that, no more scientific type tools, data science tools, that gets tricky. So we documented all that put it up on the blog post for how to do that. So we deploy our lambdas as Docker images, we build those Docker images using code artifact where we deploy the wheels for our like Python library. So the same Python code is used in the lambdas is used over in the jingoes. And it's all deployed together using the code pipeline. Actually, we aren't using code pipeline, that's one, we used to get that different customer had used GitLab this time. And actually like Git lab, that was kind of my first experience with using the CI tools in it. And I'm definitely a Git lab curious now. And once. In the past, we've used code code, pipeline and code build on Amazon or have used BitBucket Pipelines. We're currently doing a project where we're doing Azure DevOps pipelines. So we've got a lot of experience across all these different ones. And I'll say the the GitLab ones are really nice. Really impressed with that one. Yeah, that's
Carlton Gibson 36:05
good. That's that's a nice little tip. Because you've, as you say, you experience across the board. I have one more question about the details of this project, then let will jump in, perhaps, but how are you dealing with NumPy arrays and perhaps pandas data frames? I'm wondering how do you serialize those into the API? I mean, are you using particular packages to help you get from the kind of scientific Python world into the API world into something that's Json serializable? Or are you right? What are you doing?
Calvin Hendryx-Parker 36:35
Boy, now you make me dig deep into my brain there and what we did, now, I will tell you that if you are going to try and use Redis, outside of Django, in basically have them both share the same cache, it is possible that you will, you'll stub your toe hard pi the first time you do because the settings for jingoes Redis, which is where we're basically doing these kind of move this data, I think we don't store the data frames in a more native way than I think taking them into JSON data, putting that in the Redis Cache and be able to serve it back out through the the API, but there is a setting you have to be aware of on the Redis Cache side, which is to allow it to be used from other tools. Otherwise, it Django is Redis Cache functionality starts kind of doing some native II things that aren't compatible with like just getting a key and setting it in Redis.
Carlton Gibson 37:33
Okay, but you come you come straight into JSON, and then you serve that JSON as it comes. Because that's that's the sort of tricky
Calvin Hendryx-Parker 37:39
question. Yeah. For now. And it kept me we were also looking at Oh, my gosh, I can reverse Webpack, or one of the other, like, binary ways. Yeah, message back. I think the knee wasn't there yet. And it's obviously easier to keep it non binary. Until we don't have to deal with the performances problem. And then performance, there wasn't the problem, the performance was early in the in the data frames, and the pandas work against the NumPy arrays. Because some big arrays, when you think about data, weather data, I didn't know much about weather data until we started on this project. But there are just tremendous amounts of data that are generated every second of every day, for the atmosphere and all across the whole, like, we're only dealing with United States right now. But basically, you can split the whole United States into this giant grid. And then you get layered data for the various, you know, altitudes in the atmosphere. And you get all that I mean, you now you got to figure out what to do with it. Yeah, how to use it?
Will Vincent 38:41
Well, one of the cool things are the weather I was sharing with some friends recently is there's these interactive wind maps, both for the US and the world where it's same thing they're getting, you know, wind measurements, and you can see, I think they're using d3 or something like that, you know, JavaScript front end. So you can see, you know, more or less real time, what the winds doing. And so it's a very sort of accessible way to see weather and realize the other sensors picking up everything. But as you say, the question is, what do I do with it? You know, how do I compute it and how I present it? That's kind of the and the hard, hard.
Calvin Hendryx-Parker 39:15
Oh, and the lightning networks are even more interesting, because they've got Lightning sensors that can detect very accurately where lightning is striking. And so we actually use that to feed back into the system and do almost like unit tests, you know, did the productions, where lightning was gonna be, so we have a feed from the lightning providers into the system as well, that we periodically run tests against of the predictions.
Will Vincent 39:45
That's super cool. That reminds me a little bit of, as you say, there was an app, Dark Sky, which was built in Albany, New York, which is near I used to live. I think Apple bought it, but it was one of the first they did yeah, it was which is a little sad. But it was one of the first to make predictions. And it was, again, it was using that's how I first knew it. Oh, there's all this government, you know, taxpayer funded data. And nobody had thought to do predictions around it. I think hopefully, Apple is bringing that into the default Apple weather app. So I don't mean to necessarily slag on Apple, I'm sure it's a good exit for them. But it was very cool. They haven't killed it yet. Yeah, well, hopefully, they're pulling it into the core weather thing. But I remember it was because I almost I almost interviewed with them, it was I think it was like six or six or seven recent college grads, who, you know, had that insight and took the time to build it and use these predictive models based on existing weather data. That's really interesting. And, you know, as you said, lightning strikes, there's so many consumer business to business applications of that, especially if it is truly accurate, and you're testing it to confirm that.
Calvin Hendryx-Parker 40:50
That makes a lot of sense. Yeah, it's essentially you have the extra validation layer that we have data now about what we said was gonna happen did if it did happen or not?
Carlton Gibson 41:00
Yeah, that's just too much science
Calvin Hendryx-Parker 41:04
is crazy. science experiments driven
Will Vincent 41:08
by Python, actually repeatable, unlike, you know, half of all papers. And in science these days, I looked up, so when I was mentioning, struck molecules, where there's millions and millions, like millions of millions of millions of molecules around specific compounds. So I know specifically, like, like Harvard is building its own data set, all these different places are building. So they're private, but they're looking for ways to, you know, license it to pharmaceutical, pharmaceutical companies, or this that the other thing and so the structure, isn't that crazy. I mean, they basically need to put a database onto the web, but like, lockdown, you know, proper restrictions, and they have, you know, good developers in house, but it would be better if there was like an agent, I'm sure you could have an agency that just took scientific databases and put them on the web, as, you know, b2b API's. It was lots of science stuff happening.
Calvin Hendryx-Parker 42:03
So it sounds like a new startup for us all to go do. Yeah,
Will Vincent 42:07
I mean, you know, there's another one here, it's actually a big Django shop path AI that's doing this building it that's doing pathology with AI, and they're testing it against actual pathologists. So they're building their model. And you know, they're gonna have a database that they sell, eventually in use. So variations of that are, yeah, especially around here in Boston. There's it's not so much web first, but it's very heavily Python is being used science first,
Calvin Hendryx-Parker 42:31
which is kind of interesting. Yeah. is awesome.
Will Vincent 42:35
I did want to love just crossing that
Carlton Gibson 42:38
going. Well, I just wanted to ask you about like, we've kind of half touched on it. But as you know, experienced Django folks, what do we think on, you know, what can we offer to folks who like, I've got some science data, I want to get on there. Like, what? What are we missing? i What's the, if we say if we took where we are now. And we said no. But now you've we've got, you know, managed.py? Do some science. What would that do that what, you know, what's the what's the thing that we could offer? The scientific community? The Django extensions?
Will Vincent 43:09
Yeah, well, what I know, yeah, yeah, that's what I was saying. Like, you know, if they have, they just have a database built in something or other with, let's say, drug molecules. And, you know, they don't need quite drag and drop, but you just need to put that on the web and lock it down, and have control access. There's.
Calvin Hendryx-Parker 43:26
And that's, that's the hard part, right? I mean, you see a lot of really interesting, no code, low code platforms out there today. But when you kind of do more than just the Hello World, or default kind of path in those tools, I feel like they fall down a little bit. When it comes to real permissions, you start doing a lot of like, putting stuff in code into strange places to basically simulate a good authorization system in the tool. I feel like tools like Django, with the middleware and the ability to have the that control at the views. You have much easier way of expressing authorization than you do in some of these, like low code type tools.
Carlton Gibson 44:10
So would it be like a starter project or something that you know, his, you know, his name is? He's connected to your data set p that coming from R or pandas or whatever? Is a web page is an API for it is morth. Is that Yeah, that's
Calvin Hendryx-Parker 44:29
the DRF tutorials pretty good. I mean, you can go in and there's like logged in version or the anonymous public versions of the API's, I think it needs to go a step further, maybe show more granular roles than just the logged in mixin type type typicals views you see.
Will Vincent 44:48
I have to plug my book because I just added a ton to Django for API's. So there's a lot more in permissions. So here's our answer, right? Yeah, it's Whoa, it doesn't answer everything but I it's about 50% longer and it goes much more interpretive. And deployments and kind of all these things. So that's there. I was just think that's what's needed. Yeah, I was just thinking that I, if there's any listeners who are working, because I know like Harvard Medical School uses Django, like who work in these capacities and want to come on the show like, email us, we'd love to have you on and talk about this problem, because it's something I'm surrounded by here in Boston. And I don't have any particular insight on what those challenges are, you know, I suspect maybe it's almost administrative, more than technical. But
Calvin Hendryx-Parker 45:28
I don't know. Yeah, defining those rules, as opposed to how you implement those rules.
Will Vincent 45:32
I mean, cuz you have scientists who are, you know, paid to create things, and they create them. And then it's sort of like this database, but there isn't necessarily someone whose main goal is, how do we share this? I mean, they want to, they're not just sharing it, that's not government. But they may not be thinking, how do we monetize this? Or if they are, you know, it's like a custom thing with a lot of times, you know, they're selling it to pharmaceutical companies who use it for their research. I always, if a listener wants to come on, I was gonna ask you, though, Calvin, about training employees, because you're doing such cool stuff and you run agency, you know, people come in at different levels. In house, you must have some version of scaling people up. And what does that look like?
Calvin Hendryx-Parker 46:12
It's tricky with being an agency almost have to have more senior talent, we really don't have a junior pipeline. In in place, just because the way we the way we work with with companies is not the way like a staffing company would work with companies, we're not putting people into seats necessarily, or just providing them with talent sitting in a junior role. People come to us really to solve hard problems. So our our staff is really focused on heavily senior people. Now, that's not to say that, we don't want to make sure that they're always staying up to date. And not kind of just sitting back and using what always worked type things. We encourage our folks to go to conferences. So I know, we got a big group, a few of us are going to pike on I know a lot of us are going to go to Django Con this year. And that's a place where we learn and connect with the community. A lot of learning happens in the hallway still where you're, you know, running up against some folks and saying, you know, hey, I got this problem. And man, it's amazing how many problems are solved, you know, just a quick chat in the hallway, or even in a Slack channel for those things. And once a week, we do a code review with our team. And I say code review, it's really a show and tell it's really much more, I find it more fun than a code review. If someone had a code review type problem, we would we would talk about code reviewing things. But most of the time, it's, let me show you some cool tool I'm playing with or some new library I'm using on this project or some new technique or some new framework, or even some old thing that I've kind of brought back from the past and reused on this interesting problem in a different way. And that that helps keep everyone up to date, because I can't tell you how often I see like a wheel reinvented only because someone wasn't aware of a technique we've already used to solve that same problem. That actually happened this week, we're doing a lot of airflow work recently. And so there's some new people who aren't as familiar with airflow, which if you're not familiar airflow, it's pretty cool. And it's written in Django. And it's used for orchestrating, typically like data pipelines, although I've seen customers of ours use it in other ways. So having a wealth of knowledge now that we can call upon to let new people join projects who were not familiar with airflow, but to speed them up, or ramp them up quickly is mostly through just chow. So how we can share knowledge amongst the team. So we have a daily stand up, you know, 1520 minutes, where even though everyone's on possibly different projects, from different companies in different different walks of life type thing. They all talk about what maybe there is if there's a blocker or if someone else in the group who may not even be on their project can help them, it really makes that, that transfer of knowledge a lot easier and a lot quicker. So people aren't stuck and not spinning their wheels and wasting time. So that's a lot of what we do.
Will Vincent 48:52
Yeah. I mean, it's a continuing problem for everyone.
Calvin Hendryx-Parker 48:56
Oh, yeah. I love learning. I mean, I'm just a turtle lifelong learner. And I just have a huge passion for solving crazy fun problems. And so the more I can spread that to my people who are on the team, the better.
Will Vincent 49:12
So that's, is that Apache Airflow. That's the tool you're referring to. Yeah, yeah. I just don't think I wonder if we find that one. From there. Come on, if it's written in Django.
Calvin Hendryx-Parker 49:21
Oh, you totally should. The so the a lot of the open source contributions are coming from a company called astronomer. Oh, that's where the projects that's
Will Vincent 49:29
where Andrew got Andrew, Andrew Godwin got one. Yeah, that makes sense. Yeah, he's there. We should have Andrew. Anyways, so we'll just we'll just have Andrew.
Calvin Hendryx-Parker 49:41
It's a neat tool. I'm really, it's basically if you took it uses celery under the covers for certain things, or it can actually swap out salary for a Kubernetes workers, but it's a much fancier, well dressed version of celery. Like if I have got tasks that have dependencies and and you've got resource constraints. And I want to make sure certain things run and other things don't run. And it that whole orchestration of running either a data pipeline, you're moving petabytes of data across the pipeline each day, or you've got some kind of process that you need to ensure runs in a specific flow. Airflow is your boy.
Will Vincent 50:19
Well, I don't I don't know if I have any more questions. I've learned a ton. Carlton, you have any? I mean, we're close on time. Anything else bring to mind?
Carlton Gibson 50:25
I'm sort of, no, I'm out.
Will Vincent 50:30
You got your your homework last
Carlton Gibson 50:31
week, I'm thinking about everything that Calvin said. So that, that means we've must be coming up to limit because I'm sort of like, wild by just thinking about that. What you're talking about the team actually, that that's what really struck me is that the thing was one of the big topics always interest me is how you grow, how you stay in software development for the long haul. And, you know, to have that continually learning thing is what keeps it fresh for me into to have a company and build around that as a, as a part of the company culture. I think that's fantastic. So I'm just just giving the sending of those hearts your way. And well,
Calvin Hendryx-Parker 51:07
we, as I say, we built a place. I wanted to go work every day. Yeah, super.
Will Vincent 51:15
I would love to see a Django CON talk on Flash, if you're able to do it. I mean, because my you know, maybe where I'm at, in my career, my favorite Django con talks or you know, the mental health ones. And then the ones that are like, we had this really hard problem. And here's the tools we threw at it. And it's not necessarily canonical, but you know, like Django rest Knox, like, I remember really diving into that and learning about that a couple years ago, but I didn't have examples of it used being in the wild. So I was like, I don't know, like, it seems really cool. That makes sense. But I just didn't know of teams using it. So now I'm, I'm gonna go poke again at it. Because I was just, you know, so that's great. That spurs that in me hearing that.
Calvin Hendryx-Parker 51:53
I will be giving a talk on flash at Python web conference. Ooh, yeah. There you go. Me and the the lead developer from the Flash Team will be co presenting this case study. I believe I've proposed it to the DjangoCon call for papers. It maybe not even I don't think so. But it will be it will be pitched because it's a good it's a good one. I agree.
Carlton Gibson 52:15
Great. Okay. And so it takes out just remind us when the pilot about the dates of the Python web conference again. So March 21, the 25th. Okay, and what's the website? We'll put it in the show notes, but it go
Calvin Hendryx-Parker 52:28
to Python web conf.com.
Will Vincent 52:30
Yeah, well, it's everything great callin Always a pleasure. Thank you. Nice. Thank you so much for taking the time to come on. Again. We'll just have to slot you in every year. You can tell us the
Calvin Hendryx-Parker 52:39
I'll make it I'll put on the calendar for next year.
Will Vincent 52:44
Yeah, so we are Django chat.com, chatdjango on Twitter and we'll see everyone next time. Bye bye.
Carlton Gibson 52:49
Bye.