Paolo is a Django contributor and prolific conference speaker based in Italy. We discuss using PostgreSQL-powered full text search within Django as using GeoDjango for mapping projects.
This podcast is a labor of love and does not have any ads or sponsors. To support the show, consider purchasing or recommending a book from LearnDjango.com or signing up for the free weekly Django News newsletter.
Carlton Gibson 0:05
Hi, welcome to another episode of Django Chat, a fortnightly podcast on the Django web framework. I'm Carlton Gibson joined as ever by Will Vincent. Hello, Will How you doing?
Will Vincent 0:13
I'm great. Hi, Carlton.
Carlton Gibson 0:14
Hello. And today we've got a special guest, Paolo Melchiorre. How you doing, Paolo?
Paolo Melchiorre 0:18
Hi, I'm fine. Thanks for having me.
Carlton Gibson 0:22
Know. Thanks for coming on. Thanks for coming on. Super. So Paula did tell us in the listeners who are and how you got into Django and programming and Yeah,
Paolo Melchiorre 0:32
well, sure. So I'm, I'm Paolo Melchiorre and from Italy, small town, close to Rome. And I started programming in high school. Many years ago, I started with Pascal language. And then I haven't any internet connection. So I started studying myself HTML. And for the dissertation, they finally exam I, I made a website, I was the first one my my school at the time. And then I decided that web developing is my, my life. And I studied computer science at university. And I was very interested in Linux and the basis and all this good stuff. So I made a dissertation on free software, and about licensing, free software developing and then starting working for a company. with Python, I started the only the classical language in the university like C, Java, similar one. But then when I started working with this small company, we worked with plone. Yes, is CMS based on zope. And so I started using Python 2.4 in this certification, and was very good because I started also being involved with in biketown. community. I attended my first conference in Naples. prone conference was the international one in 2007. And so I'm made internet website for few years with blown. Unfortunately, it was based on the zodb itself. So object database, okay, is an evolution of the pickle from Python. One of the first no SQL database in the history.
Carlton Gibson 2:37
That's kind of cool, right? I mean, yeah, like pickled pickles. Nice, because you get some objects, you've got a few things in memory guy need to keep those, I'm just gonna pick them, and then you get them back. And they're exactly how they were before. And there's none of this, you know, serializing to
Will Vincent 2:51
dangerous, right? I always think of pickles. I think Danger, danger, danger. For security? Oh, yeah, potentially, yeah,
Paolo Melchiorre 2:58
was a bit difficult in that time to interact with the classical website or other technology, because there is only this zodb Ds object database and no one know anything about it. So you're forced to translate that in relational database. And for this motivation, I started using Django. At one point, I changed my company and we started exchanging that industrial software. So I picked Django because it was one of the first framework database it work well with Postgres, so it was Django 1.0. I remember. And other things strange was that plone was not ready for with gi. You can deploy flown on with DJI at a time. And now I think they updated all the infrastructure, but then I started using Django in a vci. environment with relational database, and I was I've fully know which had mean and with battery includes possibility for Django and then I found also that you can use very easily geo Django and postgis for for ga ga s. So and then I'm still here using Django after I think, a lot of version so this is my my trip from starting it to now.
Carlton Gibson 4:38
So you found it and you stuck with it. And here we are. Yeah, yeah. So I wanted to ask you about geo Django because you I saw you gave a great talk at Django con European Copenhagen on geo Django so
Paolo Melchiorre 4:53
yeah, yeah. And we made this
we rebuilt this website for our company. Money in France. They sell all around the world, buildings and house looks really close to the beach. So the name of the company was married mare, my friends is not good. So I don't know if the pronunciation is okay, you got to say
Carlton Gibson 5:17
that.
Paolo Melchiorre 5:18
Yeah, yeah. And so we use the boss's boss gifts and leaflets to, to shoot them up, or the only thing you can sell and you can buy on all around the world. And we made everything with Postgres from the geographic location to the full text search, in the description of these houses or around the world was very interesting, because we, we did everything in the same environment. We didn't use any external engines to search for location or search for information. And we had also this visit, search. In their website, you can see at same time, the location and how many of every option you can, you can find the clicking on different filters. So a, I wrote this talk and I spoke at Copenhagen last year was very interesting, and received also a lot of requests of information. And our we did this this platform, because a lot of people found it the tutorial into Django, documentation was a bit complex. And me too, I found that it was very, it present a very, very complex case study. And then you can instead adding a point field on your model, and starting using leaflet or something using the open layer widget in the admin. And it's very, very easy
Carlton Gibson 7:27
to just add the single field.
Paolo Melchiorre 7:29
Yeah, so we add also other field for boundaries and for states, other things. But if you need only a location for your hop check in the map, it's very easy to add, you can add both feet. Oh, of course, you have to use a JS really database like SQL lite, Postgres, also MySQL, and Oracle, but it's very easy to use you. After migrating things, you can start storing location, and like other tags or number you can use a store there.
Carlton Gibson 8:10
Okay, I'll ask you for the show notes. If you've got a recommended tutorial or something that makes a little bit less heavyweight than the Django geo Django tutorial.
Will Vincent 8:19
Okay, perfect. I think geo Django we've had Anna Kiefer on to talk about it. But I think for both of us, Carlton, I don't really know much about that area of Django. I mean, that and the ORM are the two areas where I'm pretty ignorant. But of the two I think geo Django is would be a little more fun to dive into properly.
Carlton Gibson 8:40
At some point, what I liked about your Django con talk as well polar was the the how you showed using leaflet j s to because it's like okay, so I put I put a lat long in my database, what am I going to do with it? Ah, well actually, there's some really good UI frameworks that you can just plug it into and it's quite simple and you're up and running and that's you know, you get to a certain standard quite easily
Paolo Melchiorre 9:02
Yeah, the the shortest path I think you can use it's leaflet can understand geo JSON out of the box and you can also
install Django rest framework with the
it's it's plugin you can
you can Xers geo Django as exactly as you can expose this your location data in JSON format and leaflet can understand directly and you all your know your front end developer your team can understand it without any other addition or configuration is very, very easy to use.
Will Vincent 9:53
Would you Is it still leaflet its own thing I thought it I thought the creator went to matte box and because I used matte box a little bit. Is it? Yeah, one in the same now, are they still separate?
Paolo Melchiorre 10:04
No, I really don't. So that the creator work now for my matte box. But leaflet is still
active development
library and
they are several things I think mapbox found the low point are these libraries, I think that we use also mapbox for personalized the diets. You can personalize still at graphics and corners of your map. If you need to, you can also use other types from other services. And we also use the this this stack for the last project I used the posses NGO, Django was for Mumbai application in Android. We send location to this mobile mobile application they send to us feedback with another location. So you can use your web and mobile phone and you can store everything in you know, your database in your posterous with no problem at all.
Carlton Gibson 11:14
Yeah, I mean, it's one of one of the use cases that I've had is you get the location data from the phone, and then you need to stop stop
Paolo Melchiorre 11:21
the back end to jangles ledger. You can also calculate live distances and all other teams.
Carlton Gibson 11:32
boxes, intersections.
Will Vincent 11:34
Yeah, I've done a bunch with map box, but I remember it it and maybe leaflet came on the scene around I think it was 2012. When I was in San Francisco, when Google Maps they severely. They tried to charge for their API. And so like the company I was at, we went from free to 10s of thousands of dollars and everyone around us and everyone said no. And so because leaflet and map blocks are both based on Open Street Map, which is sort of as I believe, like Wikipedia for geolocation. Yeah. So that was that was sort of the Cambrian event for at least the startup world, everyone said, because nobody talks about using Google Maps with geo Django that I'm familiar with almost everyone uses leaflet or matte box. These days.
Paolo Melchiorre 12:22
Yeah, it's true. We, we use some services from Google that are better, they're like,
Will Vincent 12:32
address 30.
Paolo Melchiorre 12:34
Yeah, yeah. Because they are better function for this. But also with OpenStreetMap, there is a nominatim is a service you can use if you don't need a so very careful location in in a city is it's good enough to search for addresses. But one thing I like, of this solution is that you don't rely on external services. So suddenly, for example, like Google Maps, they start charging or closing or limiting their services, you have to change you, you don't have all your data in your database. Instead, with Postgres and Postgres, you store everything on your own. And then maybe you can ask for, for for for other additional information externally, like, for addresses, similar,
Carlton Gibson 13:34
so your big, your big Postgres user, right? Like you, I mean, not to not to knock the other databases, but you're a big fan of Postgres. You've been using it a long time.
Paolo Melchiorre 13:43
Yeah, I started realizing a few years ago, that is the enterprise database for open source environment. So at university, we use very Holden and havy database like IBM, Oracle, it's similar. And in at the same time, that field everyone uses MySQL, because WordPress, so other PHP frameworks based in MySQL, then when they start using with Django, I found that there was still in that time. Very interesting and powerful feature. And I started also attending the posterous day here in Italy. And speaking about how we use the Postgres from the point of view of a user, with Django and with with Python, and every time I try to update my, my Postgres, searching for a new feature to use Also Full Text Search was one of them. So I think it's it's few years hides from other databases.
Carlton Gibson 15:11
So I saw you on Twitter the other week someone was, there was a thing going round about what's your tech opinion that you know, get swords pointed at you and yours was that of you running Postgres. You don't need Elasticsearch.
Which I liked.
Paolo Melchiorre 15:28
Yeah, I am a starting using photo search with Django. I found it from the release in Django 2.0. I think they really it was the search fields that you can add on. And yeah, when I found this feature, I was shocked, I realized there is other things to use instead of external services. Unfortunately, I forced to use external services for doing search in mobile application and other web services. I want to clarify, and I don't have anything against these services, because more of them are also open source and based on Lucene. It's open source from the Apache foundation. But I'm an engineer, I want to solve problem, I don't want to install other software software from other person and spend my day trying to connect solution. And to find the best configuration, I want to solve things writing code. So using the Django her RAM and relying on the Postgres fixers functionality, I was able to concentrate my my hair 14 doing the best searches begin second. And without having to have more than one instances of the external agents to solve the configuration they stuck in the middle from Django Ram. And the languages to query in the last week. I was frustrated after two or three projects using this external services. I think other people can do a better job than mine using that. But when I found that I was I was able to search directly in the database in Postgres. I started using hit a. And then I started also asking myself why in the Django project website, they are using Elasticsearch. So after studying a bit, I proposed to the Middle East to update it. And there was people against and people pros about this, this date about it, and also, the fellow of the time team, say that he also had troubles updating the version of the Elasticsearch and libraries to communicate with Elasticsearch. And so I propose a pull request, the time was 2017. And after alongs, reviews and updates, finally, we start using Django and posterous fulltext search internally, and now it's three or it's working. So I'm happy.
Carlton Gibson 18:44
And you're busy doing the same thing again now, right. So as we speak, there's a pull request open to update the version of Django that Django project comm is running and update the
Will Vincent 18:54
Yeah, version 12. I saw there was a hole, Marcus and Tobias
Carlton Gibson 18:58
Yeah, yeah. So I mean, you know, there's a, there's a bit of an ops thing is that we have to get all the various pieces into place in order to do that, but that's super work.
Paolo Melchiorre 19:06
Yeah, yeah. Because, as I shown on my tour of Django, update, every every years with a lot of features are also Postgres do the same things. So every year there is new languages they had that you can search for text in, in new languages. Also, Russia now or other different type of languages in Postgres, and they had to do also, web search syntaxes. And other feature searching collides, or they speed up their indexes. And so if you want to use all this new feature, you have to update frequently, your Postgres version, and also Django version. So I started proposing to update the Django version, the Django project. And if we at the end, we will be able to update it. I want to propose new new feature, and to the Django search because this year I asked the everytime question to user also conference also last year in Copenhagen and local meetup for feedback about the search function in Django project, and I received a lot of requests for improving the functionality or to change something to be more user friendly. And the second pull requests I did in the past for the search functionality was the multilingual functionality. Because that with elastic The only English worked for a few years, and then we Postgres, we was able to add more than 20 languages in the web search. So I think also to attract more people in the Django existem. The first thing they found searching for Django was the the documentation the search functionality, so having a better working in search function is main and main Mangle try. I mean, you know, I'm using that every day, right. So I need I mean, quickly jump over to Django project calm quickly type in the search bar, you know, I need I need. So just before we recording, now, I'd looking at the same site cookie settings, so quickly type in our becomes, you know, if the quicker the better that search works, the better. Yeah, it's so I use every day to because I use also for searching about Full Text Search, because sometimes I I forgot about all the attributes you can use, or it's impossible to remember everything and I prefer understanding outing things work and then reading the details in the documentation, because it's, it's very convenience. I
Carlton Gibson 22:13
think a lot of people find out about features in databases or in Postgres, in particular from the Django Doc's. Right? So you find out about yellow fields or JSON fields or re fields from reading the Django Doc's.
Will Vincent 22:24
Yeah. Well, because I, because Postgres had full text search for Was it 10 years, it was quite a while before. Yeah, it made its way into Django itself. I was, I remember will link to your talks, and you have a great article, I gave a talk, was it last year on search. And basically, I was just looking at your information non stop all last year for inspiration. So it's really excellent. But I remember, because I have a little bit less programming experience in both of you and I came to Django from some other frameworks, the two of the things that jumped out at me is that there wasn't a built in signup, author registration, I had to do it myself. And that search was also lacking. Because a lot, a lot of other frameworks have some something or they have elastic bundled in. So I agree that ways that we can make it easier and have an out of the box solution. I mean, it's kind of the whole point of that talk, and a tutorial I have was showing, like the layers of search, and you can quite easily do basic search. But then, of course, it gets much more complex. So I think that's a broader thing that's good to have accessible to Django, people, or people coming to Django, who are used to just having built in search in whatever framework they're coming from. But I love that you're pushing the boundaries with Postgres itself, because we had on the founder of listen notes, which is a site that does searching through podcasts, and he's using Django, and he's also using elastic, because he was also familiar with elastic. So the fact that you're an expert, but you're also not just porting over, you know, existing expertise, you're pushed, you're still learning and taking advantage of the new functionality that's being added. That's not always the case. But that's obviously extremely important to have someone doing that and talking about it, rather than the confines of the enterprise system where they maybe can do that.
Paolo Melchiorre 24:15
Yeah, motivation, because I try to promote chango and pasquesi. fotosearch is also because plastic is very famous, and they do a very good job in marketing and out of project are using elastic, so they don't need other people speaking, produce the solution and not saying anything against the solution. But I'm only saying that if you need an easy and quick solution to implement a full text search, and you're already have Django imposters is inheriting in advance. So it's the
Carlton Gibson 24:57
thing you've already got it running, you're already running Postgres. So what I take on extra difficulty that extra complexity the extra obscene use that first,
Paolo Melchiorre 25:06
there are Yeah, I think scenarios when you need external search engines. For example, if you have centralized the search functionality with Django application and other no Java application and no SQL database, a relational database at the same time, if you need centerpointe to search on, maybe you can think start thinking to the dedicated search engine. And it's, it's convenient in this case, but I work in medium to large projects in my company, and usually a positive is enough for everything. So if you have more than one particular replicated, you can manage millions of records. There is a lot of space. And if you need elastic, maybe you you have a very big project, otherwise, it is not. The example
Carlton Gibson 26:13
that the listener notes founder gave us was high interest rates, because he's ingesting updates to podcasts, and they come in like, you know, there's a lot of podcasts. So if you're trying to index all of those, the updates a lot, and that was his, the reason he cited for needing Elastic Search was that he needed to handle that inches. Right. But most a lot of sites on like, well, I saw that Apple, I think, just in the last month or two added the ability to search, like within podcasts, which so he's got a fantastic service and hopefully he'll he's got enough of a running start that he can keep that going. Or maybe they'll just they'll just buy him. But
Will Vincent 26:51
yeah, he's at a crazy scale. I want to ask about so hosted solutions. So algolia I'm just curious if you've played around with algolia because that's something I've used a bunch and seen it develop. I just curious if you have any personal experience with that option.
Paolo Melchiorre 27:08
No, I am here to say golia Hi he is for past project for on startup you use the sort of like Elastic Search is based on Lucene and I use I tried also Sphinx further another project, but in every new every project, there was dedicated instances of this services. In one cases there is a an external company that money HD is external search engines and we are I was forced to do all the search on this search engine or and hold the rights in the database on our databases 70s and was very frustrating because there was synchronizing problem. And people have this
delay in
searching for information and accessing this information because maybe they modify a detail on a project of a project. And the writing in the database was very fast. But then there was forced to wait for the synchronization to the external search engine. And so the the final user experience it was very, very frustrating. Because they are asking where is my data? And say you have to wait because the external search engine is very slow or there is a connection problem as eventually consistent. Yeah.
Carlton Gibson 28:51
Where did my data go?
Paolo Melchiorre 28:53
It's funny my also my wife usually fine for non shopping online. And every time they complain because they found the last one shoes in the market when they click it when she click on found that in the today page, their shoes is it's gone is your right there is not anymore, and I explained to her is because they don't synchronize their belief of products and the detail page. So it's very, very common problem.
Carlton Gibson 29:27
Yeah. And from the perspective of you're crafting your own Django application, that's a difficult problem you want to postpone for as long as you can. It's making sure that on every right to your database, your in your search index is updated, if you can avoid that problem. Yeah, you know, forever put that off as long as you possibly can because it's
Will Vincent 29:44
a big one. Well, and and the issue is search, of course, is that an average user is used to Google so anything on your average website that isn't Google, they're gonna say, Well, this is crap. Pretty high bar. You're also a Django girls coach, and there's been a change in leadership, I believe we're gonna have one of the new founders on quite soon. I just want to ask you about maybe you could tell people what is Django girls and in your involvement with that community?
Paolo Melchiorre 30:15
Yes, of course, for the first time, I
was a coaching the Django girls workshop in 2017, in the Euro, Python. And it was a very interesting experience. Django workshop was one day workshop for, for girls organized all around the world. And they accept girls that want to learn how to write Python and Django and working in web environments. And you can propose yourself like a coach. And during these days, you can coach one to four girls, and Epping them to following the steps in the Django girls tutorial, is an online tutorial, it's very easy to use, you can also use on yourself. And usually I suggest also to people that want to and following something more longer than the Django tutorial. And you spend the day answering for question about why Python is not installing good in Windows machine or very old Linux machine is not it's not enough for similar question his own
Carlton Gibson 31:38
files on getting that kind of thing.
Paolo Melchiorre 31:40
And when I, my first coach experience, there is two girls very, very secure, they already use Linux. So I waited all day, asking if they need some help, but at the end, I was not able to have been there because they are already very, very skilled. So I updated the tutorial to Python 3.6 In the meantime, to spend the day, but also we organizing in Rome. And immediately they are very active, they organizing, you know, CDs around Italy, but also all around the world. They there is every month workshop, unfortunately, during this period of pandemia, it's difficult to organize in person worship. So I saw people organizing also online, but I hope that after this periods was gone, we can start to gain organizing Django girls workshop days and in conference or during the here because it's a very good.
Will Vincent 32:51
Yeah, it's an awesome program that's there reminds me there's I need to be a I need to be a coach. I thought about it, I haven't done it. And certainly once things are normalized, because here in Boston, there's a very active Python community, which has regular meetups and even workshops. And that would probably be a perfect place to, you know, have have a number of people who are because part of the problem with Django, you know, because I teach I teach it for a living is you have to know a whole bunch of other things before Django makes sense. So I like I spent a lot of time with you. Okay, after install it. Have you installed Python? Do you understand web development, you know a little bit about HTML, CSS, there is that issue of what's the bare minimum you need before you can dive into Django? That? So if you're at a Python conference, presumably they have a little bit of Python, right. But if I just walked into a school, I can't just probably start teaching Django.
Paolo Melchiorre 33:44
Yeah, and this location is good, because there is girls from every environment, people from the vertising people from a school. And you found out de, and this assumes chango from very different backgrounds. And yeah, I learned a lot trying to explain to this girl out Django. Why Django work in that way and not in other other way too. And also, you can meet other coach and have a very good reaction in your your city or your region with other very skewed and, and people that care about the community and also involving girls community. I think it's very good.
Will Vincent 34:39
Yeah, no, for sure. One. I'm thinking there's also a Django meetup in Boston, which I think is one of the biggest ones in the world that I don't think we've done a Django girls there, we should do that. But the other questions you get from people really solidify or makes you think, right, because people come in, maybe they don't know anything or maybe they come from an MVC, setup and They find controllers confusing or? Yeah, it really makes you 360 on what is Django and how is it different? But I think that's something actually I know this pandemic will probably continue for a bit. I want to do some videos to explain just like, how did the pieces fit together, like because Carlton has a fantastic talk on middleware. But I think middleware is something that your average giant Django person doesn't really understand at all. And yeah, but it could be, you know, I could just show them Carlton's talk. But you know, it could be simply explained, and you know, peeling back the onion, but it's, you know, explaining, explaining the interaction of views, templates, models and URLs. That's basically the Django story if you can internalize those four things. But that's kind of the Django way, I guess. Anyways, yeah, people should volunteer in you'll learn a lot. You meet great people, as you say,
Paolo Melchiorre 35:50
yeah. I opened the next Django con Europe. We will have a Google workshop to
Will Vincent 35:57
maybe well, they just they just announced they're gonna try to do it in I think it's June. They put it out. Yeah.
Paolo Melchiorre 36:04
So I started the week of June, I think.
Will Vincent 36:06
Yeah. So I guess that happens. I still have my plane ticket that I haven't gotten refunded for. So I'm ready to. I'm ready to go. Yeah.
Paolo Melchiorre 36:17
The last Django con. European Copenhagen was very good. Meat also cart on there. And also the spring today's was very, very interesting, emits so many talented programming, Python programmers, and is a very good occasion to speak with our people exchange ideas. I think we would be able to do again.
Carlton Gibson 36:47
Yeah, I miss it. I miss it. The sprinter Copenhagen was where I realized we needed a workshop for new contributors, because I was there trying to help people and I was running around different tables, I must have spoken to 4050 people, but there's only one of me is so I wasn't able to do that. So in Django con us San Diego afterwards, I did a workshop we had like four tables all together or work. And it was brilliant, because people were helping each other. So Jang. Porto was meant to be like the second running of that workshop. But it obviously didn't happen this year. So
Will Vincent 37:20
next, Carlton, you're working with? I mean, from Google, we got a season of dogs. Grant? Yes. There's work being done on the contributing section.
Carlton Gibson 37:30
Yeah, yes. As the season dogs project is going on that will be running till March. with Carrie precious, who's from Nigeria. So she's working on on that now. And hopefully, that will make the contributing guide more accessible. Because, you know, starting as a fellow I was asking, you know, you know, why don't more people contribute? And the big answer I heard a lot of times was, hey, the contributing guide is really quite intimidating. Like it's, you know, several thousand words long, and it's got all the information you need in there, but perhaps it hasn't got a pathway, which guides you into that first contribution as well as smoothly as we could. So that's what we're hoping to improve over the next few months there.
Will Vincent 38:08
Yeah. Well, so we're coming up on time a little bit is, is there anything we haven't asked you about? Or any projects or things you'd like to talk about?
Paolo Melchiorre 38:17
Yeah, let's see, I think interesting. In the in the company where I work, we use only Django and want to say that this year, the Nobel Peace Prize winner was the World Food Program. And we did work in Django for the World Food Program. So Ah, Django is involved also in in this fields, and I'm very happy to, to say to other people to, to make they understand that open source and free software can connect the humanity in general, in various form also writing code. It's a way to talk to other people, and it's a good way to, to be in human beings. Super.
Carlton Gibson 39:06
Yeah. So Django should Nobel Prize winning web framework
Will Vincent 39:10
for this year.
Carlton Gibson 39:13
Yeah, Barlow. Thanks for coming on that super What a great What a great chat. Thank you so much.
Paolo Melchiorre 39:17
Sink you hole in this particular day to speak with me and for remind me is very good space.
Will Vincent 39:25
Supriya Thank you.
Carlton Gibson 39:26
We're Jango chat. Join us next time. Bye bye.
Will Vincent 39:29
Bye bye.