Anna discusses GeoDjango, a built-in contrib module that turns Django into a world-class geographic Web framework.
Carlton Gibson 0:06
Hi, and welcome to another episode of Django chat. I'm Carlton Gibson, and I'm joined by Will Vincent Halliwell. Hi, Carlton. And today we have Anna keitha with us. And we're gonna talk about geo Django. So how are you?
Anna Kiefer 0:19
I'm doing well.
Carlton Gibson 0:20
How are you better? Well, I think we're marvelous. But thanks for coming on the show. So yeah. Awesome. Let's kick off. Perhaps a good way of going if you could introduce yourself. Tell us about your background, how you got into program for instance, and how you discovered Django when such things like that.
Anna Kiefer 0:38
Yeah, yeah. Actually, so I got into software development through a bit of a roundabout way. I was not sort of formally, you know, trained as a computer software engineer. But I was working at World Wildlife Fund on their renewable energy team and was tasked with helping sort of design build, you know, a simple Just website for some of our corporate renewable energy partners, and that's where I got introduced to software development. And I really love that. And so I did a fellowship in San Francisco. And, you know, that gave me a little bit more course more foundational knowledge. And I wanted to combine energy, my sort of interest in energy in the power grid with programming and software engineering. So building web applications, you know, sort of track and measure the energy grid, and I guess the physical world around us. Yeah. Yeah. And so I started as a software engineer at Kabbalah analytics, and they're an energy analytics startup in San Francisco. And they do a lot of mapping so you know, mapping feeder or distribution lines. substations, transformers, solar panels, so mapping all and aggregating all this energy data, and that's where I started learning Django and using geo Django.
Carlton Gibson 2:11
Okay, and so were you what were using before to self interest? Like,
Anna Kiefer 2:15
um, well, I don't think we had we weren't really utilizing any, I think just Postgres. Right. And so doing Rob, sort of, you know, using psycho PG to or something. And connecting to Postgres that way, and not really, you know, very hack. techie. I guess, not really using that interface. But you were you were lucky enough to start with Python then. Yes,
Unknown Speaker 2:39
yes. Fantastic.
Anna Kiefer 2:42
Yeah. Yeah.
And so yeah, when I came on board on to Kabbalah, we were they were using already using Python and are using Django. I don't think they had started utilizing geo Django.
Carlton Gibson 2:57
Okay, so tell us about JJ. What is what is to
Anna Kiefer 3:01
Yeah, so geo Django is Django sort of answer to geospatial mapping. So it's their geospatial web framework. So it's, it's just a contrib module basically through Django that handles geographic information.
And it offers,
you know, the ability like Django, to you know, have models and classes. But in addition, it offers the ability to handle and manipulate geospatial data, transform it,
Carlton Gibson 3:42
what's it what's, what's when you say handle geo, geo? geographical data, what do you what sort of things am I gonna do with it? What sort of uses Am I gonna have,
Anna Kiefer 3:53
right? Yeah, so I think maybe the easiest way to describe this maybe let's take like, you know, type of data. You want my Want to model right? So say a road. And so just like a regular Django class or model, you might have, you know, a road has a name a road has maybe a length road has maybe the state that it's in all this sort of, you know, metadata that are just strings. But a road also has a geometry associated with it. And so geo Django allows you to specify a geometry field. And this can be a point line, a polygon, you know, multi line or multi polygon. I think there's like eight different types, something like that. And you basically specify this sort of road, you know, road would probably best fit as a line. And through that, through that attribute, you're able to do transforms, you're able to Do other sort of geospatial and distance queries?
Carlton Gibson 5:06
Yeah. So could I could I could I create a TomTom type app? Like, you know, if I had all the roads in is locating yourself
Will Vincent 5:13
with that reference? Yeah.
Carlton Gibson 5:16
What's it what's the TomTom with Google Maps?
Yeah. Can I do Google Maps? Like, turn by turn direction? I guess it's right. Do something like that.
Anna Kiefer 5:25
Uh, yeah, we did not, you know, we did not build some something at all like that. Well, you know, we had a map and we had but but yeah, that and other some other fancy, you know, algorithms and optimizations, you could in effect, create google maps where you enter in, you know, the address of your destination and geo located your specific points. And using geo Django you could route to that point, you know, the fastest way or something. But Google Maps is quite pretty, right. It's got those nice tiles and it shows me the pictures, but how do they do that with geo Django
Carlton Gibson 6:02
I can't use Can I use Google Maps with it? Or do I?
Anna Kiefer 6:05
Yeah, you could use Google Maps API certainly
with a, you know, using geo Django as the framework back end framework. You know, so So what the user sees on the, on the front end, right is just, you know, the Google Map. Those are like Google Maps tiles. Of course, Google Maps also uses like all the API layer and all the fancy things but but you could certainly use you know, say leaflet or matte box with like different actual tiles that are say that Google Maps tiles and that is just the How do I describe like the I guess what you're what you're seeing what your skin on top of the
Will Vincent 6:47
right, right top? Yeah, yeah. Right. So I was, I was just gonna say so I've used leaflet and matte box. I was curious for you, and how would you How would you describe that is the set the options available? Because I mean, leaflet is open source map box is open source data though they're VC funded. And then Google is their own thing. I think as Google, I'm remembering I'm dating myself five, six years ago, Google's did one of their periodic we should monetize our API things for Google Maps, and charge like, hellacious rates and then everyone got all excited about kind of matte box and these spray source so so what's your take of the landscape right because I know now there's Apple Maps yeah yeah, I guess it's a different was another one that box uses that data right? I think are you right? They were a skin on top of so I guess the two parts are there's like proprietary data which like Google and Apple don't share and then there is OpenStreetMap which I believe is what powers map box and leaflet Yeah, and leaflet is more of a open source whereas map box is also open source was raised a ton of money and right
Anna Kiefer 7:52
right I think ultimately map boxes This is the umbrella over leaflet and leaflets the matte box sort of free. Oh yeah. They hired the leaflet PR guy, as I recall.
Will Vincent 8:05
But anyway, so how do you think about this with projects? Right? Because I simply break it down into like, yeah, there's Google API's, but they could charge a lot. And there's open source. What are the what are the factors you think about, like for a personal project, like cuz you have some really cool ones on your site, which we'll link to, versus an enterprise setting. Are there any distinct right there that you would make? Right?
Anna Kiefer 8:27
Um, I would say ease, you know, ease of use, and that's where leaflet and map box to me have really made it like their documentation is is really good. They're way better. Yeah, I mean, we were actually at Kampala, we were using Open Street Map, and not using leaflet or you know, or matte box. And we found it a little bit. You know, they're, they're a team of I think their volunteer. You know,
Will Vincent 8:54
so it wasn't genuine can do it like you and I could go in and it's sort of like Wikipedia, right? Right.
Anna Kiefer 9:00
Which, you know, in some regards is really, really great. But I guess they were less Full Feature than we wanted. And so, you know, leaflet just has so many sort of bells and whistles, if you want to create a map that looks really nice. It's got the ability to coroplast which are just overlays. Yeah, it's got, you know, really easy way to code. Let's see, you know, satellite toggle pop ups. You know, all these sort of the things that you know, a user would expect to, you know, to see on a map.
Will Vincent 9:42
Yeah, can you say a little more about chloroplasts because I, those are, people haven't used like, hopping that's a big word would explain that further. I've used it but I you know, right. But first, right. Yeah, what's an example right? Like, I mean example would be something like No color scheme for a dead like you, you have predicting disease spread is one of your projects. Right, right. rhetor right, that would be an example of a clip. zactly
Anna Kiefer 10:09
Yeah. So, so it's it's a, you know, way to visually, you know, display data when, say, points or lines don't convey the information you want. So something that's like disease spread would be really, you know, a good way of sort of maybe
or where where you would use a choropleth
that way, you know, areas that are really
affected by the disease. There's a really dark or dense
in areas that are not
Carlton Gibson 10:49
so it's like a temperature map on a weather forecast,
Will Vincent 10:52
right? Yeah, yeah. Or like a political like a red blue, purple, like us political kind of map. Okay.
Carlton Gibson 11:00
All of this is built into leaflet. j. s. Yeah. Okay. And is that part of geo Django? Or do you have to pull it out? Pull it in for your front end when you're writing your templates,
Anna Kiefer 11:09
right. So that's not part of geo Django. Exactly. When you, you know, you, you basically, after you, you know, construct your view, you throw the sort of leaflet, you use the leaflet API to just pull a map up in the browser. Yeah, and display your tiles and then you send your geo JSON, oftentimes from Django, to you know, display on the front end.
Carlton Gibson 11:39
Okay, and how do I serialize so is that so say, I've created a Django model, a geo Django model, and I've got some, you know, write geographic data in there. I've got you know, my roads or the the shops in my town where I want to go shopping next week. I've put that into geo Django for some reason. How do I send that to the front end? Do I need to use rest framework or can I just send data geo Django Give me the serialization tools I need just to send
Anna Kiefer 12:04
me the same way you would send data through through Django. So you know, you would use a music SV use dot p y. It's been a little while since I've used Django. But yes, you use your views.py and then your URLs. And then you use geo Django to you know, once you've retrieved your data from whatever database to serialize that data as geo JSON. Yeah,
Carlton Gibson 12:31
and then, but the point the question was, the include geo Django gives me the serializers. Yes. To turn right. Okay. Yes. Sometimes you need to put in a third party like rest framework, or I don't know what
Anna Kiefer 12:44
Right, right. I believe it gives you the serializers you need. Yep, fine. And along with you know, I think the biggest benefit or the biggest Yeah, the biggest benefit, I guess that that I liked using you Django four was that, you know, you didn't need to do these sort of hairy SQL queries, you know, raw queries in your Python, it provides an interface to post GIS, which is, Postgres is, you know, geographic database, to do these perform these queries,
you know, much simpler. And so,
you know, Aquarius, they taking that roads example, say you want to find, you know, all of the roads that intersect with a hospital or that, you know, run into the hospital or that are within you know, a certain distance of your hospital. geo Django provides a really easy way of filtering and doing these distance queries. Whereas with, you know, just with SQL, it's, you know, a little bit more
confusing, I guess.
Will Vincent 14:00
Yeah, I mean that's sluggo rM, right? Yes. from raw? Yeah. Well so so you had a fantastic talk on geo Django which we'll link to which I think I Carlton I both came to know of you. But can you recall what was the process of? Like how would you recommend someone learn geo Django because it is somewhat advanced use of Django and these other technologies like how would you baby step someone up that curve if you could do it again? Cuz I suspect you were kind of maybe thrown in the deep end or did you start more on like the front end JavaScript side? Or what was Yeah,
Anna Kiefer 14:31
I started actually with flask. As many I think Python users do, you know go from Yeah, and then thank you know, a little need a little bit something you know, something with a little bit more, I guess bells and whistles and and structure
Will Vincent 14:43
batteries.
Anna Kiefer 14:44
Yeah, yeah. And then you know, I just remember for the first project I did, I was banging my head against the wall I was trying to get Oh, it was trying to get I think it was I did this project where I mapped landfills across the US. Cuz I wanted to I wanted the user to be able to enter their zip code, and then see where their trash you know, routed to. And if that
Will Vincent 15:09
landfill was full,
Anna Kiefer 15:11
you know, where how close to capacity that landfill was. Anyways, and I remember banging my head against the wall, trying to, you know, figure out, you know, how to get this data to just display on my titles. In I was probably something very minor that I had done wrong. And, you know, all of a sudden, the entire US was full, you know, I made must have made one tweak, and all the US was full with landfills, you
Will Vincent 15:38
know, yeah, I mean, mapping was quite realistic. It's hard. And then you also you have the visual component. So it's not like just like a little, you know, error. It's just like, like, you can see the mistake, right.
Anna Kiefer 15:47
Yeah, yeah. Yeah.
And oftentimes, you know, there's, there's simple things like the lat long, you know, switching those Yeah. Yeah, yeah. So geographers systems often I think nearly all the time actually post GIS. Certainly geo Django, they use laun longitude and latitude to define a point, whereas users are a little bit more familiar with latitude, you know, and then longitude fall right and xy makes more sense than one Right, right. Anyways, yes so so how would I get started with geo Django so Django does have this great tutorial on you know, seeding a database with some geo JSON or with a shape file. And, and then it walks you through, you know, the process of querying that data and manipulating it a little bit. What you could also do is just if you don't want to worry, you know, don't want to deal with you know, maybe setting up a database connection. You could just, you know, import God go and grab some geo JSON, you know, in a one, you know, in a Python file and just yeah, you know, use the methods that way, that's really not using all of the things that geo Django offers. But it would allow you to say, you know,
Will Vincent 17:20
very simple sort of, sort of queries, because you could pull in like a geo JSON of like a map of like the US states, right? Or just to like, sort of explain how, so geo JSON Excuse me. But I mean, like the few polls, so if you had like a discrete data set of geo JSON, is that kind of like it would just have like the outlines of the you might get it right, like there was,
Anna Kiefer 17:44
yeah, so.
Will Vincent 17:46
So gj would you import right? Like we're like, Where did you get it? And what would it actually look like before you applied? clerk laughs chloroplasts to core plus, right, right, what plus plus I'm saying it wrong. I think
Anna Kiefer 17:57
it's coroplast but I always forget it. What else? Yeah?
Will Vincent 18:01
Yeah, like chloro or core is that it sounds like very Yeah, scientific, right. Anyway, yeah, that that sort of didn't really make any sense. But I guess I was asking, if you think of so you have your data, and then you have how it's displayed? And then you have, or how does geo JSON differ from the skins that we've been talking about, like leave like leaflet and matte box? Right? A lot of times you need to have the geo JSON yourself or get it from somewhere, right before you can apply that skin on top. But it's different from the coordinates that are would be in the database itself with Right, well, your Django database.
Anna Kiefer 18:36
Yeah, so so a leaflet or a map box or some other you know, sort of front end web framework like that allows you to display this data and so the geo JSON or the shapefile, or whatever, you know, that is the actual data that you care about, that you're going to put on the map and you're going to use matte box or leaflet to You know, get it on your map and to display it. And so geo JSON is just one form of geographic data. And this is, you know, geo Django and post GIS allow you to transform, you know, you can turn your geo JSON into a well known tax to run on binary.
And,
you know, and use it that way, as well.
Will Vincent 19:28
Right? Well, I guess I'm trying to think so like if you had like a table of geo Django data, so rectangular, and then on top of that, you could get the geo JSON or your your shape files. And so then you could say, if I'm within the shape of a US state, you could map and see is that does a data point in my database fit into this particular shape? And if so, apply some logic to it and just try it like verbally. Think about how long The pieces stuck together for people never use this before. Because it's all confusing.
Anna Kiefer 20:04
Right? Yeah. So so you would have
you know, you're at
Will Vincent 20:09
the bottom.
Anna Kiefer 20:12
Yes, the bottom would be just the data, right?
Will Vincent 20:15
But coordinate data with whatever nit data separation. So look at lat long, like diseases 19 out of a scale of 100 or something.
Anna Kiefer 20:25
Yeah. Yeah. Yeah.
would be Yeah. Just your, your data, and then, you know, you, you would put that data, you know, into a data table. And again, you could assign a, you know, geometry field
to this data.
That's just done in Django classes
Will Vincent 20:52
over it, but that's the one there's a couple options, right for how you would do that. Right. Is this like a modal class so
Carlton Gibson 20:57
yeah, like that. You've got your normal Django model. line if you didn't get disease disease rate, and then you want to say it's at this particular location is that so you could add a point field would that be?
Anna Kiefer 21:09
Yeah, yeah. So say you have, you know, let's say illness, instances of illness or something, right. You know, you could have, yeah, you could have, you know, one instance of illness would be a row in your table, and then you would define, you know, as, you know, a column that's, that's that lat long at that location.
And that would just be that would just be be a djenka point field.
And then you'd be able to, you know, take a polygons, say, from a different table, maybe you have your state's table, your US state, and, you know, do a query to combine these two, so you could say, how many instances of illness are within this particular state?
Carlton Gibson 21:56
And then then you show it on the map.
Anna Kiefer 21:58
Right, right. Yeah. And this is where, you know, geo Django is. makes this all very easy, fast. Super.
Carlton Gibson 22:07
Okay. And a couple of times you've said post GIS, which is Postgres, which is like the Postgres extension that enables all this geographic stuff, right? Yeah. Yeah. So. So geo Django can be set up, I think with a number of different databases. MySQL SQL Lite. spatialize is like the the sequel lite version. Yeah, that's just a file that and one might think that's the simplest to get started with. But is that the case? Is that Is it simplest to get over that or simply as with Postgres, or,
Anna Kiefer 22:38
I guess so in the sense that you don't need to I think that's what Django comes automatically with SQL Lite. If I'm, like, Yeah, well, that's uses it as sort of its default. But it's quite easy to get started with Postgres as well.
Will Vincent 22:53
I think it's a you need to add either spatial light to SQL which is built into Django or if you install post, gray SQL, you need to add Post GIS on top of it.
Carlton Gibson 23:02
Right, right. It's the installation, right? It's the so what would you recommend people to do?
Will Vincent 23:07
Which are the two?
Carlton Gibson 23:09
Or the three, you know,
Will Vincent 23:10
well, well, Oracle is not supported. And then my sequel was it's a bit of an issue, I believe, but you're the expert. So you tell.
Anna Kiefer 23:20
I've actually use primarily Postgres and post GIS. So I would have to say, you know, use those, but I don't want to make any enemies.
Carlton Gibson 23:30
I've heard tell it's the most capable as well.
Anna Kiefer 23:32
What is post GIS? Yeah,
Carlton Gibson 23:34
yeah, I've heard. I've heard people say that. It's the it's the more advanced, it's the
Anna Kiefer 23:39
Yeah, it's, you know, they've.
They've done so much with geospatial data. We were, you know, at my current company, were doing some of these queries by hand using go a different you know, not Python. And the They're, they get very complex very quickly. So using something like geo Django, and you know, and post JS really just make these querying and filters so much easier. Leave the hard work to, you know, to people who've already done it.
Carlton Gibson 24:21
Yeah.
Will Vincent 24:23
Because the idea yeah, I'm sorry, go ahead.
Anna Kiefer 24:25
Oh, no, I'm sorry.
Yeah, I was just gonna say various
you know, things that geographic data can be, I wouldn't say finicky, but
more like it can be easy to get
incorrect, particularly with units.
When you're doing filters, you know, is the meters or miles there's something. there's a there's a geography concept and then there's the geometry concept and geo Django also You know, reconciles the two of these, but a geometry concept takes the say polygon or line takes the shape and models it on a 2d plane. Whereas the geography uses a spherical representation. So that's sort of, you know, where you the state comes in, right? So it's taking, you know, these points and actually models them to coordinates. And this concept, you know, you need to make sure to transition between the two at points, or else, you know, you might be, you know, your units can get off, get all messed up,
Carlton Gibson 25:40
and so does do Nanga do the right thing there. And, you know, if you pass a geometry into a geography it says, No, hang on, you've really got to convert these furs. Yep,
Anna Kiefer 25:50
yep. And you can specify typically, I think geometries are a little are more common to use, but you can specify that you want to use a geography and then it will It will handle all of that for you. You don't need to, you know, be doing transformations and crazy sort of, you know, x y to, to, you know, to your projected coordinates, all that, right. So
Carlton Gibson 26:15
the more listen to you The more I think, Wow, actually, there's just massive masses, masses being wrapped up in masses of maths that you don't have to know masses of SQL that you don't have to write. Like.
Anna Kiefer 26:27
Absolutely, absolutely.
Will Vincent 26:29
Yeah. So what's the goal equivalent, right? Cuz I assume it your current job, the ideas that go is faster in quotes than Python, and therefore, you can't use all these nice things. Is there a framework for go spatial data,
Anna Kiefer 26:45
there are some
there are some I'm trying to think of the one they're not they're nowhere near as popular as
Will Vincent 26:56
best say something like geo Django or So you have to stifle the urge to say, you know, this was solved if we'd been using what I previously used.
Anna Kiefer 27:04
Yeah. In fact, worse. We're switching back.
Okay, are you really back? Believe it or not? Yeah. And part of the reason is that we were doing we were creating all of these all these functions by by hand, basically, I mean, we're using the the go GIS libraries. But even with that we needed some custom functions that weren't available or aren't available and go. So actually, yeah, we're switching back to using to using Python using post GIS, which we could have been, we're certainly using with go as well. But most of the database remains the same database remains a different framework on top of it. Yeah, it's more the, you know, sort of Python, geospatial libraries that are really full featured, that we weren't quite getting as much as in go. Again, I don't want to make any enemies and say that there aren't great libraries to use. But
Carlton Gibson 28:02
that was kind of interesting because, you know, on on on the on the chip go is faster, you know, it's a faster language is more low level, it's, you know, it's got higher throughput, but there's developer time as well. And you know, if you're if your Python is fast enough, the developer time Trump's it Right,
Will Vincent 28:20
right. Right. Well, I mean, as we talked about often to hear, generally in a web Doc, the language isn't the big issue. There's a whole lot of other things that are slowing you down.
Anna Kiefer 28:31
Yeah. And we found the development with like I said, the development with go and with creating these all these geospatial, you know, functions ourselves to be a lot just, it was costly developer time. And we also aren't inch aren't sure we're doing it right
Carlton Gibson 28:49
all the time. Right. Okay.
Anna Kiefer 28:50
You know, you can write your tests for your geographic function, but you know, you can make your test pass the the beauty of tests, you know, you also write them who knows what the test But it's still may not be entirely accurate. So
Carlton Gibson 29:04
yeah, it's so sometimes a an algorithm matches a test case. Okay, all the test cases, but then yeah,
Anna Kiefer 29:10
the both air.
Will Vincent 29:12
Right? Well, that's why you want always want to check, you know, does it do what I think and also doesn't do this other thing that it shouldn't do because sometimes it just always passes round. Well, so, so day to day like what so what does it look like? What are the challenges that you wrestle with? I mean, maybe it sounds like maybe classic data science where a lot of it is a little bit of algorithms and testing, but a lot of making the data fit and not be wrong, right? Because if it's 1.1% off, it blows everything up. Is that accurate? or How would you describe? Like, would you describe yourself more in the data science side or more? Some other area?
Anna Kiefer 29:45
Right? Well, I would say so my previous role, I mean, it kavala Well, it was a smaller team that I'm on now. Which was awesome because you've got to work totally doing everything. You know from front end. A lot of mapping and core uplifts. To the API layer to, you know, the data ingestion and data pipeline and to to our, you know, sort of energy developing or energy methodologies. And now I am on the data engineering and sort of a little bit more on the data science side of things. And, you know, I'm no longer doing, I guess, the the front end mapping, although we certainly have a need for, you know, for a fully fledged, full fledged front end and map framework, but I'm not currently doing that. So I'm doing more of our data science production using the data science methodologies to do things like
create a power network
using you know, machine learning. So figuring out, you know, where feeder systems and feeder lines are on the grid from, you know, from from actual data, feeding it into a machine learning model, and then validating it and seeing, you know, how close we were using
Will Vincent 31:11
the actual data. And I assume that's, I mean, that's very Python based most Yep. Which packages are you using day to day? Which which of the? Which Python ones?
Anna Kiefer 31:23
Yeah, you know, right now we're not using a whole lot and I almost wish we were using something a little bit more like I like the I am spending my time now doing a lot of these raw SQL queries.
Will Vincent 31:39
In my pipe you get to do an outer join right there's a discussion like mine, Simon Wilson, some others were saying, you know, like they've never in their career done an outer join. Oh, really? That's asked in every
Anna Kiefer 31:49
Yeah, every interview where you but you
Will Vincent 31:51
never know if it's 20 year career, Simon said maybe once or twice used to.
Anna Kiefer 31:56
Oh, that's crazy. No, I would Yeah, I do. Well, I mean, there wish it were just an outer join right? It's like an outer with you know various various other like this you know get Elva the feeder lines within a certain distance, but then mash them to the buildings, you know, and then do with some of the aggregate on the buildings, you know, so it can get pretty
Will Vincent 32:20
hairy. Is that is that something where do people talk about like graph qL in this context at all? Given that you have, you know, go like not right. Oh,
Anna Kiefer 32:30
yeah, yeah. So, before I was on the data team, this data engineering team only been on the team about maybe a month or two. I wasn't go land where it was graph QL. We were using g RPC and yeah, and graph QL.
Will Vincent 32:45
So postpone the decisions on the relations until later, right? Yeah, I just kidding. Sorry. Okay. Now, B, it seems like especially the data sets you're dealing with, I mean, that's sort of it It makes sense to maybe use a non relational in that context.
Anna Kiefer 33:03
Right, right. And we actually used D graph, which was a sort of a very, I think it's small, I think it's a team of like five, five people. And it's a, it's a graph database written in go. So we were using that, and it's very fast. But again, it didn't have its syntax is, you know, I guess we're also used to SQL and the SQL syntax. And this really kind of turns that on its head. And so we were again, we were spending a lot of developer time just trying to get this syntax working to query the database. I think with with, you know, say with practice and more experience and as the D graph team, you know, that that software becomes a little bit more mainstream. We might go back to using it but for now, you know, given All of the geospatial data that we do deal with, we wanted something like that we were comfortable. Yeah, being sequel.
Carlton Gibson 34:10
It's interesting because like a graph database, I've always found it deeply fascinating, but just never really found been able to make it work in the real world compared to the established stuff. It's like, yeah, everything's a graph problem. If you look at it in the right angle, and right, surely, we should be able to solve it this way. And then I can use these nice little pathfinding algorithms. Yeah,
Anna Kiefer 34:31
yeah. I know, right. But I was really interested in to use it was really happy that I got, you know, a couple months experience using it, even though we're not using it in production now, but yeah, we kept you know, it's it's a whole sort of mind shift, because we kept saying, you know, I kept saying, Oh, yeah, so there, you know, the rows in the table, then you're like, Oh, wait, but they're not they're not rows, right. In fact, it's not a table, you know, and you're like what? Okay, the nodes then we kept having to you Don't change our how we how we describe this data, you know, the nodes in the, you know, in the graph, right. And it's just like, a different conceptual way of thinking of data.
Carlton Gibson 35:10
Right. But it's kind of interesting. We talk, you know, in software design manuals, always talking about matching the the, the mental model of your users. In this case, like you're the user in my database isn't matching your mental model, an incongruence. And, yeah, that's kind of cool.
Anna Kiefer 35:29
Yeah. Yeah. And I think something. I mean, I think, like, we're still trying to find how to use graph databases, because some of our problems are really more suited to graph databases. Like we have something that we kind of call dependency chain and that is a link of sort of cascading of, I guess, dependencies, right. So if you think about a power system, you know, power system has different tendency say, you know, on water maybe so a water plant, let's just use this as an example. And your hospitals downstream and that has a dependency on both the water plant and the and the power system. And so you know, you've got these sort of cascading dependencies in modeling that in a table could be pretty hairy because you know, you've got your, okay dependency one, but many, you know, there's many, it's a lot of many to many relationships going on. And something like a graph, you know, can model that a lot.
Will Vincent 36:35
A lot better. Yeah, as soon as you have one many too many. I start going right. Here we go. Yeah, yeah.
Anna Kiefer 36:42
Yeah. How many Association tables you need.
And that's something we haven't really addressed. I don't think yet. At least at my company, you know, there's technologies that have but
Will Vincent 36:53
right well, I mean, the thing about data science, right is at least from I was just looking when I was studying this this list. A couple years ago, there was this twitter feed Big Data Bora, which had all these pithy sayings. And I mean, I remember being struck, I spent almost all my time cleaning the data, and then a teeny amount actually running an algorithm or something I made, and it's just kind of like it felt very, you know, janitorial, but I think that's just kind of how it is. Right? You toil? Like, it's hard to clean it, and you have to do it the right way. And then, yeah, I guess you go down. Yeah. So it's, yep. There's some quote about that.
Anna Kiefer 37:28
Yeah. Yeah. And in fact, in both roles, and I think, you know, we deal with a lot of sort of students, you know, who also deal with geospatial data. And there's, while there are certainly uniform, you know, this geo JSON is uniform ways of handling geospatial data, and making sure that you know, the data is, is the same from one government, you know, one organization to another, a lot of the format is not the same. And so, we build An entire data pipeline for, you know, R Us data, right? So that's ingesting US Census data. But we have the need now to ingest Japan census data, and it's entirely different. So that data pipeline, you know, we need to change drastically in order to get it to work with, you know, these, these files, basically.
Carlton Gibson 38:20
Yeah. And I always get the impression that data pipelines, you know, I'm more of a web jockey. But, you know, data pythons always like the sort of one shot scripts that I write for myself to do stuff where they're sort of bunched together with duct tape. And oh, you know, I'll just run this year.
Anna Kiefer 38:37
I was good to go. Yeah. Right.
Will Vincent 38:41
Is that your question? Is Is that accurate? Yeah, yeah.
Anna Kiefer 38:45
Yeah. You know, at least at both companies I've been in we've we don't, you know, it almost seems like once the data is in, right, you do it once, almost, you know, and then you don't have to do it. Again. We're not talking machine learning models or anything. We're just talking static data. So you know, you have all this. Yeah, the duct tape and you have, you know, it's all shoddily constructed, but then it's in, right? And you don't really and every time you're doing that, you're in that process. You're going, Oh, gosh, we need a better way of doing this. We need a better way of doing this, but then you know, and you kind of forget.
Right? And then, you know, then that's everything. Yeah, right.
Will Vincent 39:19
Yeah. Well, you don't want to prematurely optimize it, right. I mean, right. tooling got to use it.
Anna Kiefer 39:24
Yeah. Yeah. And so like, right now we are developing you know, well knock on wood, but we're trying to develop a you know, a more flexible, fully featured data pipeline that we can use when we deploy say, you know, in Japan and China and India, you know, where it can fit with you know, I still don't know how this is gonna work. Just Just being the one to you know, build the duct tape the pieces together. Can Yeah, I'm questioning how we'll find a solution. But exactly that's what makes it fun.
Carlton Gibson 40:07
Yeah. Programming.
Will Vincent 40:09
Yeah. Well, I think that's how often do these does this change? Right? Like cuz cuz some data sets you have to adjust, you know, minute by minute if it's like power meter and then summer. I don't know, you know, monthly yearly.
Anna Kiefer 40:20
Yep. Status every 10 years. Right. And so, you know, if you only have to do this, you know, hacky construction, right? You're like, that's not so bad. You know, we don't want to spend Yeah, but but exactly, but something like, like pricing like, energy pricing data is like every cent or no, every couple minutes, or meter data, like you're saying is every couple writes.
Will Vincent 40:47
Yeah, because there's some in the Boston area. There's a couple companies they're using Django to Yeah, like to Internet of Things to check on. Right? So solar meters. And it's, as I recall, I mean, I remember being like, Why don't device manufacturers do this. But basically, they need a degree of separation. So it's Yeah, I won't name names. But it's it's interesting. It's a very interesting problem. And it's kind of shocking that it isn't addressed by big behemoth that I'm aware of sort of spy places that are right in this mind monitoring service for these power grids. Yeah.
Anna Kiefer 41:20
Yeah, I think it's, um, you know, we're still at the forefront of all the sort of smart grid and grid optimization work. And it's gonna be exciting to see, you know, how, see how it shapes up because we have this old infrastructure. And then, you know, you're trying to glean insights and, you know, figure out basically, you know, how to optimize, you know, power on the grid.
Will Vincent 41:47
Right? Well, it's I mean, Internet of Things is I would think Carlton like a pretty good use case for you know, async Django stuff. I mean, depending, again, you think, but yet, we write async anything, right? I mean, it's a side effect. You know a chatbot or something it's it's that's a real world application when I think of async up because Django is adding async in 3.0 that's the first thing that comes to mind for me all these Internet of Things we have something just pounding off data you know every second or faster
Carlton Gibson 42:16
yeah so whenever you whenever you need anything to handle an event an unknown interval then async becomes perfect so Internet of Things is perfect you know when it doesn't know when you go and open your fridge door but when it when it gets the fridge to run it needs to handle it and tell Amazon to send you some new meal cool. Interesting.
Will Vincent 42:38
stat unexpected part. Yeah, because otherwise you could just plan for it but it's the spikes that right? Yeah, is a sink really shines
Carlton Gibson 42:45
below I'm gonna come back to the energy optimization because that's fascinating. There must be I mean, I read a story A while back about Google. And obviously they control the environment in their data centers perfectly but they use their AI to, you know, save 60% on the power bill or something. I don't know the number Right. Can we are we gonna be able to do similar with the with the grid? You talked about the old infrastructure? Obviously, we can't just rip that up and start again. But do you think there are savings to be had and gains? And?
Anna Kiefer 43:11
Yeah, yeah, definitely. And figuring out.
Yeah, there's certainly cost savings, there's energy savings.
And not being, you know, energy experts don't
say too much. But,
but there are certainly,
like, we have very little information, other than, you know, what your utility bill right, the average consumer is very little information about what the demand is on their, you know, on the grid at a given time. You know, maybe they do want to, you know, less than their energy bill in some ways or, you know, put solar panels on their roofs and, you know, if you wanted to change your energy provider, you know, basically areas have pretty much one energy provider in that That's it. Right? Like we've got a pg&e Yeah, utility, Genie, and we can't, you know, we can't change it. And, you know, I'm sure lots of people would like to. But I think giving consumers a little bit more insight into, you know, there's sort of energy and electricity, whether that's through something like the Internet of Things and the smart grid, you know, that's gonna be really revolutionising the industry.
Will Vincent 44:29
What's that? It's a two parter of, you know, you could mail someone an automated monthly packet saying, Here's your usage and how it's changed. But then you need that second part, like here's a non pg&e. So all right, right? When in the solar context, you have an interesting thing we have with people who are net contributors to the grid, and actually it sort of messes things up for the utilities if you're in a good way, like if there's too many people giving stuff because electricity actually has to kind of flow like water like in the same way that we need to like create all the farmland because they're stealing too much. Anyways. What is there anything we haven't covered that you want to address in the podcast? We're sort of coming up near the end. Or anything you want to plug?
Anna Kiefer 45:10
Right, right. I don't think so. I mean, I mean, I would say, yeah, I've, I've absolutely been very happy using geo Django and for anybody who hasn't played around with it. And they say that people who like maps really like maps. And geo Django certainly makes that, you know, makes, at least made me sort of love maps and geospatial data and doing queries and sort of making all of these data discoveries.
Will Vincent 45:37
And you're moving back to Python world a little bit.
Anna Kiefer 45:40
Yeah, yeah, exactly. Exactly. Yeah. And I think that's about it. I did write some notes. I'm trying to see. Oh, yeah, one thing that I just on the the sort of the future of of geospatial data, if I could say so. One thing I haven't handled much is, you know, I've handled mainly to do geometries, but handling 3d geometries, I think would be really interesting. So, in my current company, we have, you know, say we, we might map a flood, right.
And, you know, a flood has a depth
and actually, you know, being able to map this depth
you know, we might glean other insights that we certainly might not from 2d data
Carlton Gibson 46:32
so you know, volume of water flood prediction as well. You know, like, if you on this volume of water will this race housing estate will be under threat and it needs that kind of barriers protected? That kind of thing? Exactly.
Anna Kiefer 46:43
In fact, my current company one concern does exactly that. Okay. Yeah. Yeah, so, and I know post GIS does have the ability to handle things 3d data. You know, but I haven't played around with that too much. And but I would be very curious to
Carlton Gibson 47:07
okay, but that's on the horizon. that's available too. So that's kind of cool, right?
Will Vincent 47:12
Yeah. Yeah. That's Yeah. I mean, I have to play around with this a little bit. It's been. I haven't used Django when I'm doing mapping. Well, I haven't done anywhere near that functionality we can so I'm sort of excited. Yeah, again. Yeah. Well, so we will link to you have a personal site. And yeah, thank you so much for coming on. Thank you also for your talk last year. I thought that was a great
Carlton Gibson 47:35
talk. And I guess we should link to the docs to
Anna Kiefer 47:38
Yes, absolutely.
Will Vincent 47:40
Yeah. It's funny that there's a you know, specific tutorial because there's not that many tutorials within Django itself. People just think of the poles tutorial, but yeah, Django is really its own kind of World Within Django.
Anna Kiefer 47:52
Yeah, yeah, I think they use spatial light in that tutorial, and they use a shape file. So I think they have you sort of seed your database using shapefile and then create a, you know, a simple model to play around with the data.
Will Vincent 48:06
Yeah, that you need, then you need a way to display it.
Anna Kiefer 48:09
Yeah, I'm not, you know, not remembering what map mapping framework they use. But
Carlton Gibson 48:15
I know thank thank you so much for coming on and telling us about geo, Django and your the work you've been doing and the discussion around all of that. It's been super.
Anna Kiefer 48:24
Yeah. Awesome. Awesome. Thank you for having me. This has been great. You know,
I wasn't sure if I'd be
you know, have not having a taking a little break from geo Django, so
Will Vincent 48:35
I'm glad it went, Okay. No, no, I think that's that's the best one. You You have a little perspective on it. Right? Because we're in the raids, you don't really think think about it in the same way as when you're a little bit far apart from it. So Right, right, totally. All right. Thanks again. All right. Bye. Bye. See you guys. Bye.