Carl is a Software Engineer at Instagram and a Django Core developer. We discuss his contributions to Django and how Django and Python are used at Instagram today.
This podcast does not have any ads or sponsors. To support the show, please consider purchasing a book, signing up for Button, or reading the Django News newsletter.
Will Vincent 0:06
Hello and welcome to Django Chat a fortnightly podcast on the Django web framework. I'm Will Vincent joined by Carlton Gibson. Hi, Carlton. Hello Will. And this week, we're very pleased to have Karl Meyer join us. Welcome. Thank you. Good to be here. Thank you for coming on. Again. Thank you for joining us. Maybe Could you just quickly mention your backstory? You're a big name in the Django world? But for those who don't know you, how is it that you've come to be on our podcast?
Carl Meyer 0:29
Well, it's good question. Um, I don't
Will Vincent 0:32
know I Besides, we asked you.
Carl Meyer 0:35
You emailed me and asked me to come on. No, I, I guess I I started a company in 2008, with my siblings, doing a web application development consulting shop. And we settled on Django for back end stack. And so I started building sites in Django, and often, fixing bugs in Django was more interesting than building whatever the client needed to end up spending time doing that and got into Django development that way, I guess, originally.
Will Vincent 1:10
So what, what version of that? I mean, that was that pre one, when you first started getting involved,
Carl Meyer 1:15
it was pre one, let me think it was post magic removal, this isn't gonna really be one for the old timers here. It was pre queryset refactor when I first started using Django isn't the point point nine, seven days, I guess, which lasted for quite some
Carlton Gibson 1:34
time? Was there's still some people on it,
Carl Meyer 1:38
probably.
Carlton Gibson 1:42
And then you made your finger, right? Because like, so I, you know, acting as a fellow as I do now, like daily, I'm trawling through the history of track, and very often I'll come across, you know, comment by comment. You know, how did you get involved in the made in this sort of maintenance and running, just just contributing, just just fixing the bugs, as you said about?
Carl Meyer 2:01
Yeah, just contributing, and, I guess, having opinions I attended. And it happened in 2008, when I started the company and started doing more with Django. I was living in northern Indiana, so just an hour or two away from Chicago. And, as luck would have it, pi con was in Chicago that year 2018 and 2009. So I said, why not? I headed up to Chicago and attended pi con for the first time in 2008. And so I've been to everyone since that actually happened. And like this year. And yeah, so I guess I I attended a number of Django sprints at pi cons for a number of years in a row and got to know the Django team and I just get more involved.
Will Vincent 2:51
Okay, what I think it was 2014 when you join the technical board, was that the inaugural technical board?
Carl Meyer 2:57
Yeah, that was the first technical board.
Yeah, I think I think I joined officially joined the Django team in 2010, maybe. But then we had kind of a first big project governance restructuring that I am Rick Augustine, spearheaded in 2013 2014. And that resulted in the creation of the technical word structure.
Carlton Gibson 3:23
And so you, you must have been involved in the discussions around the creation of the say, for instance, the API stability policy and these kind of decisions where we don't with the deprecation policy, where you know, if we are going to change something, we bring it in, we deprecate it two releases Later, we'll, we'll take it out. And that, that's that, from my perspective, that's working phenomenally well. Especially compared to the early days where, you know, I've using 1.2 and upgrading, and I was always, it was always really difficult. Right. And so perhaps you could give us some light in that discussion. Oh, because it for me, it's just like a Wow, it's but it's kind of why Django is what it is at the moment.
Carl Meyer 4:00
Yeah, my memory is that that was kind of always the intent from the beginning that Jacob and Adrian kind of set that set us down that path. I think, at various times we've been more or less successful in in maintaining the backwards compatibility that we intend to maintain. But I think that was one of the reasons that one Dotto took so long to arrive because there was I guess this idea that one Dotto was the moment when the the backwards compatibility guarantees would really kick in and and so there was a hesitancy to actually declare one night Oh, we want to make sure we had all of everything in place that we wanted it. And after a while, it became clear that that was kind of a fool's errand that API's will never be perfect and we were already trying to maintain backwards compatibility even pre one Dotto because in practice, it was a problem for people if we broke it. So at some point, we just said it's time to do one. I don't know. Yeah, but I don't recall a lot of changes or discussions around changing that policy, I could be forgetting something, but memory is, we pretty much always had that intentive giving a couple of releases before changing something,
Carlton Gibson 5:11
because Okay, that's interesting cuz I would, what would I say? I'd say it's a different world now from say 1.5 1.6 1.7, where things were still changing quite a lot. And you know, for good reasons in Django grew with that. But, you know, comparing that to say 1.9, or 10, one point 11 into 2.0. It's, it's, it's very easy to update now. And is part of that do you think is just the maturity of Django? Or do you think,
Carl Meyer 5:36
yeah, I would put it down to the maturity of Django, mostly, I mean, I think backwards compatibility for a widely used library in Python is always tricky, because nothing is really private. And as much as you might try to maintain backwards compatibility, it's very hard to know what people are relying on. And I think in the one dot o through one dot seven, one dot eight era, Django was changing very quickly, and there were major new components being added. And I think that just made it hard to always maintain the the backwards compatibility that we intended to. And I would guess that if it's if it's significantly better, no, it has a lot to do with just the core parts not
Carlton Gibson 6:16
needing to change as much they don't change. Yeah, they don't change. Okay, cool.
Will Vincent 6:20
Well, the circle back to pi cons, I was watching for the second time, pi con 2012. Talk I think you gave on Django testing, because there aren't actually that many testing Django talks. And one of the things you said in there, which I thought, which stood out to me was write less view code, and maybe don't even test it with unit tests. Just do with it with functional integration tests. I wonder, does that still hold up for you? Do you still stand by that? Because I love the line, you said something like, anytime I see a view of more than 10 or 12 lines, it's sort of a smell test? Because I feel that way. So I'm just curious, you know, eight years later, does that sort of general advice still hold for you? or How have your views changed?
Carl Meyer 7:04
My views changed, no pun intended on testing in Django?
I don't know, I still write short views. So my views haven't changed?
I think. Yeah, that's still basically what I believe in these days. I know, it's been a number of years since I've really written a Django view, to be honest. So in practice, it's not that particular question isn't an issue for me. But I still certainly believe in pulling, pulling business business logic out into testable components and isolating it from IO, in order to make things testable. And that's essentially what that advice boils down to.
Will Vincent 7:45
Yeah, no, it's fantastic advice. And I'm, for me, I'm thinking of doing it in depth Django testing course, at some point. So I'm trying to review to the extent their best practices, because that's one of them use PI test. But it seems kind of all over the map to me, honestly, aside from I don't, I don't see any pure, canonical way to test in Django. But yeah, I'll find it.
Carl Meyer 8:09
Yeah, it's interesting. Coming to Instagram and seeing sort of what happens in a very large codebase, many years later. It, it certainly highlighted some of the some of the things in early Django testing frameworks that maybe weren't ideal, like the emphasis on the test client, and on writing lots and lots of kind of end to end functional tests. And, you know, it's easy to say that that doesn't scale. But seeing in practice, how it doesn't scale is another thing entirely.
Carlton Gibson 8:44
You met, you mentioned the big guy, word. So I have to ask, so you can't you came to Instagram. It was built on Django? Yes. And then, you know, for me, this is, you know, super interesting topic, because I've never worked on anything like that kind of scale. But I've worked on some big sites, and people say, can you scale Django? No, yeah, of course, you can scale Django. But then there's a level above that. So I guess, you know, what, what? Can you sort of talk us through that the evolution of the, to the extent you can for Instagram, and how they hit those scaling problems, and that, you know, the approaches that are needed to take and things because it's something that I think most of us will never see.
Carl Meyer 9:24
Yeah, that's a great question. I gave a talk in 2016 at Django, under the hood in Amsterdam, where I really kind of go through a lot of that history in detail. So it'd be an interesting thing to look up if you're curious about sort of the scaling challenges as they came. My memory from the research I did for that talk is that as you'd expect, I think the ORM was one of the first whether some, some kind of funny minor things that hit earlier on, like turns out sending an email to this site maintainers every time there's a forum for Scale particularly well, that became evident early on. But But yeah, as far as bit sort of big components, I think the ORM was the first one, there was a, there was a need, I think, after about a year already to, to be able to shard the database. And at that point, it was still pre multi database, I think. Not sure multi database support might have happened around that time, but I don't Instagram certainly didn't have access to it and and there was no real support for, for charting databases in the origin. So Mike Krieger, the founder of Instagram, basically sort of wrote a sharded ORM, kind of a very barebone started ORM and moved from the Django or into that, and then later from that to tau, which is a graph database built at Facebook, which is what Instagram uses for the primary data store. Now,
Will Vincent 11:03
I recall, I don't know. Here's an anecdote, I'm not sure where I heard it. But I believe it to be true that in the early days, there was a Postgres issue with the space allocated to users was so big, that they couldn't figure it out. And I think Frank at a rep says, there was some hack at the time, but helped them with but just these crazy scaling challenges, and I think it was just, you know, the two founders and pushing the limits of software at the time. Yeah, one of the funny stories.
Carl Meyer 11:33
Definitely people told when I first arrived, Instagram was an early days, once Instagram sort of really kind of made it mainstream. And I think Justin Bieber was one of the first big Instagram users. And because of the way that the sharding scheme was done, essentially, like a user and all of their media are shorted together. And which makes sense for sort of access patterns. But it caused a bit of a problem when you have sort of very popular users, like Justin Bieber, would make a post and then everybody, you know, goes to like, and comment on that post. And all of a sudden, you have a ton of database activity, all concentrated on one shard. And I apparently caused this was before my time at Instagram, but apparently that caused a lot of problems and sort of memorizing Justin Bieber's user ID was a requirement for Instagram operations for a period of time, because sort of every time he posted made a popular post, it was sort of an operations emergency that everyone had all hands on deck, I think
those things are in a lot better shape now.
Carlton Gibson 12:43
Yeah, I guess I could make similar stories about Twitter, who, where they got popular users, they sort of pre populate that separate thing was that the average user you will not you will meet but like just a small, small people, you'd fetch them on demand, because they're not manufacturers.
Will Vincent 13:02
So Python, Instagram, you're wearing a Python Instagram shirt. So they're still using Python, but they're not using so much Django can. Can you talk about where the stack is today in terms of Is there any Django left in there, and how much is Python as opposed to other back end languages?
Carl Meyer 13:20
I mean, Instagram server is still a big, Python codebase. Big, it's, it's Django at the core. I mean, it still runs through the Django request handling stack, and the middleware and views and URL routing, all of that is still Django, it looks familiar to any Django developer. Just very big, lots of us lots of URLs, lots of code. There's a lot of I mean, obviously, the ORM, we don't use anymore, the admin is very much based on the ORM. So that's not in use anymore. And there's a lot of components, where Django provides kind of pluggable back ends, like sessions and authentication. And for all of that stuff, we've essentially rewritten the entire system. But we're actually still using the sort of Django API for those things, because it's pluggable. So I would call that a success story. I mean, in that, we've been able to very smoothly migrate away from the components that that no longer worked at our scale. And in many cases without even having to change the client code touching those subsystems because Django provides this ability to swap out the swap out the back end. So yeah, and you know, the Django core requests support that we still use, we're 100% async. Now, so we've had to already a few years ago, we kind of forked and modified a bunch of that to support async, or concurrent handling of multiple requests using async. io. So there have been some changes, but
Carlton Gibson 14:54
it's kind of cool that you still got the core requests in there because I gave a talk at Django con Last year that the one that happened, not this one that didn't. But talking just about that, that color, and there's only like 160 lines in bass handler, right? There's nothing there. So there's no, the point I made in the talk was there's no room for it to be slow and clunky. I mean, that sometimes gets described as going down. But when it's not doing anything, it can't, you know, it can't be still hold doing that same thing that all the rip frameworks are doing. So that's kind of that's really nice. I was wondering what would Instagram at the scale, they're at need needed to have moved away from Python to a, you know, a faster language, quote, unquote.
Carl Meyer 15:33
I mean, you know, there's certainly discussion of that.
Python is not particularly efficient. Now it uses CPU, and there would be more efficient languages. And that would certainly save on server costs. But there's also a lot of code that would have to be rewritten. And that also has a cost. So there's certainly always a debate about the pros and cons there. So far, the conclusion has been that we'd rather work with Python and try to improve Python and make it more efficient, rather than trying to rewrite all the existing code in a different language.
Carlton Gibson 16:10
Okay, that's cool. You said you said you said improve Python there. So can you? What might that mean? I mean,
Carl Meyer 16:19
yeah, well, I mean, that's the team that I work on now is kind of actively looking at the Python, language and runtime and where we can tweak things to get a little more efficiency out of it or, or more developer efficiency. So one of the projects we've talked about publicly is something we call strict modules. So like one problem that we have with Python is the fact that when you import a module that can execute arbitrary code, which can do anything, it can go out and talk to the network, it can end up being very slow, it can fail. And when the code base gets very large, and there's really especially you can have all these side effects of import. And sometimes one side effect of importing one module can be a prerequisite for some side effect of importing another module. So they have to be important in a certain order. And that situation can arise even accidentally over time. But yet, the order the modules are important, isn't really fixed or determined anywhere, it's kind of a emergent property of the entire system and all the imports and all the modules. So you end up with this very kind of fragile state where somebody can remove or add an important sub module in one part of the system. And that has a cascading effect of changing the order of imports. And then something can fail just because two modules are being imported in a different order. So another side effect of that or Another consequence of that is that if you make a change to a Python module, and you have all of these modules loaded and running in your web server, say you're a developer working on the codebase, you make a change, there's really no safe way to say, let's just reload the one module. And sort of hot patch, this one change has been made, there's, there's nothing you can do that's safe, other than start from scratch, start the entire process over again, and begin the entire process of importing and loading all those modules. And you know, when you could be because very large, that takes 30 seconds, 40 seconds, 50 seconds a minute, which is a long time for a developer to wait for making it simple change to one view, before they can test that view in their browser and see the effect of their change. So something that we're trying to do with strict modules is do some static analysis, to restrict what is allowed at the top level of a module to say, you know, at the top level of a module, you can define classes, you can define functions, you can use decorators, as long as they don't reach out and have external side effects, basically, like you can do all the things that we normally associate with Python, at the top level of the module, but they have to be contained, the only effect they can have is within that module. And it once you can guarantee that that's the case, that unlocks a whole lot of possibilities, because now you can just safely re import modules, it doesn't matter what order modules are imported in all of these problems go away. So that's one, one kind of example of a project we're working on. More in the developer efficiency side of things.
Carlton Gibson 19:20
Okay, that sounds really cool. When you talk about the static analysis, I mean, how does that tie into the kind of thing you see with the type hinting that is coming in? And like the PI, Lance thing that Microsoft have got that doing the static analysis there and my PI related tools? Is it? Yeah, it's clearly it's the same ballpark. Right. But you're using you're leveraging those.
Carl Meyer 19:44
Yeah, for sure. Yeah, it's I mean, so strict modules in particular, doesn't make too much use of of, of type hints, but it's definitely all in the same kind of area, which I think is a really
productive and growing area right now. Figuring out how can we
do more advanced analysis and make stronger assertions about what Python code does and doesn't do without actually having to run the code to find out. And so type hinting is one example that we've been very excited about we, we started adding type ins to our code base, I think already in 2017, maybe 20. I think 2017 already. So at this point, our server code base is basically 100% type, everything has type annotations. And we've found that to be a big productivity boost for developers, and makes the code safer. And so we're looking at, you know, how to further leverage type hints and, and static analysis to make Python both faster and more efficient for developers.
Carlton Gibson 20:48
Okay, can I ask a follow up then, because we had a discussion about whether we would bring type hints type anything into Django itself? And we're the conclusion that the technical world came to work will not right. And now because that is difficult, and you know, that the technology is still nascent? And we have, so if you could give us some advice, like, what would you say to you know, if How is it feasible? Are we going to be able to get Django there? What should we be looking at? Can we start small and grow in what just as you're sort of, you know,
Carl Meyer 21:20
yeah, I'm a big fan of, of type Python, I think it's an improvement to the language. But I think that the decision that technical board made on that was a reasonable decision, probably the right decision, it is very early days, still, I mean, at a place like Instagram, we can kind of go all in on something that's still a little rough around the edges. And our developers are kind of willing to put up with some rough edges in order to get the benefits of typing. But for a framework like Django that is so widely used, I think it does make sense to wait until the the technology has matured a little bit, and some of the rough edges of usability are, are smoothed out. So I think I think that was a good choice, I think, you know, it's always a challenge to bring static typing to code that wasn't developed with static types in mind. Because patterns are used that just don't lend themselves to static typing. So I think, you know, it's been fine for us at Instagram, because you know, we don't use the ORM. There's parts of Django that are particularly challenging for static typing that we don't use anyways. So for us, it's been pretty easy to just have some stub files that cover the API's that we use, and, and that pretty much covers our needs. As far as typing Django itself, it will be challenging to do a good job of, of typing all of Django, especially given the backwards compatibility constraints. The other thing is, when we're typing Instagram code that was written in a very sort of dynamic way that doesn't lend itself to static types. You know, if it's internal code, we can just rewrite it, or, you know, change the pattern. And for Django that's much more challenging because of the backwards compatibility constraints. So
I think it'd be great. If someday, yeah,
Carlton Gibson 23:12
you can say very, for this code, you have to pass an integer, not int, or any.
Carl Meyer 23:17
Right, exactly. We can just, you know, go through the entire code base, and change all the call sites and make sure they all do the right thing and just ship that as a diff, you know, you can't ship diffs to all of the code that uses Jango in the world. So
Will Vincent 23:33
the question I have is, so you talked about what you've been working on recently, but it's been, I think, five, five years or so since you've been at Instagram. There's always this, I always have this mystery of when someone like you, so Django core developer goes to a large company, it's a little bit of a black hole in terms of like, what do they do? They're like, how does it work? Do you come in and they say, What do you wanna work on? Or do they say you have to do this? Like, how, how does that evolve? Right? When you? I mean, you're focusing on, you know, eking out performance in Python and advantages for developers. But yeah, just curious what your story was like. And if there's any sort of standard paths when, you know, a well known figure goes into a large company, and then isn't doing open source work all the time.
Carl Meyer 24:14
Yeah, I'm sure it varies by company. So I can only really speak to my experience in general. And this would apply to any engineer, not not just someone who's coming from an open source community. Facebook and Instagram have a very kind of engineer driven culture. So you know, there's a pretty wide latitude for an engineer coming in to pick the teams and the projects that are interesting to them. And then the culture is sort of you, you decide what you want to do, and then you have to show that the things you did had some benefit, rather than somebody telling you what to do. So there's a lot of opportunity in that it can also be a little bit scary but But yeah, so that was my experience coming in was that there was a lot of freedom to say, Well, this is, this is what I'm seeing here. This is what I think is important for us to work on. And so I'm gonna go work on that.
Will Vincent 25:10
One of the public facing things I saw, I forget how many years ago is that there was a small team that switch from Python two to Python three, over the course of I believe, was only like three or so months at Instagram, which was perhaps an example of people wanting to make a change taking responsibility. But I'll link to that post. I mean, that was, it's pretty amazing that even at the size of Instagrams, codebase, back then, a small team was able to do it relatively quickly.
Carl Meyer 25:36
Yeah, that was very impressive. That was one of the first projects I worked on, it was already kind of a, it was kind of it had just been started when I joined. So it was kind of something that was already happening, it was easy to jump in on, and I thought it was important for us to do so. I worked quite a bit on that, on that project. Um, yeah, I mean, it felt like it took us a while, but I think it was under a year, which is impressive, given the amount of code that we were talking about.
Will Vincent 26:01
For Django, may this is a prosaic question. But so what version of Django to the extent that there is Django and Instagram Are you running? Right now?
Carl Meyer 26:09
Yeah, unfortunately, we're still on one dot eight. So when I joined in 2016, we were on Django, one dot three, which was already quite old at that point. And we upgraded to one dot eight, the biggest motivation for that was to allow the Python three upgrade, because Django one dot three didn't support Python three. So we did the one eight upgrade, we did Python three. And, you know, the problem at this point is just given the amount of Django code that we use, you know, there's some big deprecations that happened in one dot nine, one dot 10, things like string views, went away, string view names and URL configurations went away. And there's a bunch of new restrictions around like, the import of models, files. Anyway, some things that would be a fair amount of work for us to adapt to, and there just really isn't a lot of corresponding benefits. Cuz, I mean, you know, I don't think changose development should be oriented around the needs of a company like Instagram, it's, that's not the 90 or even 99% use case. And so the things that are being done in Django, by and large are things that we either already have our own solutions for just aren't gonna be relevant to what we're doing. So I think at this point, given the forks we've already had to make for things like a sink, the most likely path forward is that we end up pulling in the remaining Django code that we use and just kind of continuing to iterate on it. It's kind of our own internal thing, rather than probably ever doing another Django upgrade.
Carlton Gibson 27:49
Yeah, that's kind of it's clearly going to be the case, I guess, as you say, it's not even the 99%. The five nines out, it's like the one in a one in a million. But it's also kind of sad. It's Yeah.
Carl Meyer 28:07
Yeah. Yeah, no, it's definitely been one of the one of the sad things about the way that my work has evolved over the last four or five years has been that I'm very disconnected from Django development these days, I don't even really have a clear sense of what's in the recent Django versions, because I'm not getting to use them, obviously. So that is a little sad. But I'm still having fun working on Python. So it's all good. That's fair
Carlton Gibson 28:35
enough, you'll have to start a side project. I wanted to ask you then. So if you're not going to contribute to Django and work on Django anymore, what about other things, open source things? Are you able to do open source work, or
Carl Meyer 28:49
I've made a made and helped with some contributions to Python itself, since small things in the standard library and
a few other patches that we needed. One thing was
we had kind of a silly issue with the way that our development process works. The local development server, we actually run you whiskey, which would run in production and also for local development, but it runs as root just because it kind of legacy of how the operational systems developed. And so it was, if you enabled PVC file creation, it would sprinkle all these PVC files owned by route through the code tree. That obviously doesn't really work. So at some point, years ago, before I even got there, someone just turned off pcyc file creation for local development. And at some point, after I got there, I was looking at this problem of how long it takes to start up the server and local development, when someone makes a change how long it takes to restart it. And I was like, well, we have no KYC files. So not only are we re importing all these modules, every time someone makes a change, we're actually like re compiling them. from scratch, you don't even have Pac files. And that's costing us like 25% time, which is really silly. But then there's there wasn't really a great solution to that. So I just wrote a feature for Python where you can set an environment variable, and say, put all my Pac files over in another directory somewhere else. And just kind of kind of a parallel directory tree elsewhere, of all your Pac files, which works great for us. Now, we can do that we get the benefits of byc file caching. So that's just one small example of like, and then yeah, I contributed that upstream. I think it came out in Python three dot nine, maybe three days, I think three days or so. So yeah, a few things here and there with Python, but not as much as I would like. Okay.
Carlton Gibson 30:44
And do they? Do they give you time? I mean, like, I don't think they still do it anymore. But Google used to have this 20% time or whatever that I don't need still there. But did you get that kind of thing? Or is it only if it's project related? You don't have?
Carl Meyer 30:58
Yeah, there isn't an SEO kind of a explicit 20% time or anything like that. It's kind of like I mentioned earlier, like anything you work on here, it's like, you can do what you want. You just have to, it's just to say, Hey, I did something useful this app. So it's kind of up to you how to how to allocate your time. Okay.
Will Vincent 31:19
That's cool. I'm curious about feature flipping at scale. You know, in Django, there's like, I like there's this project called Django waffle. That's really great. But Instagram is a whole different beast. Are there things? What things crop up, you know, living features, like, I think I've seen Instagrams and playing around with removing the like button in Canada or a whole bunch of countries? Like, what does it even take to roll out something like that?
Carl Meyer 31:43
Yeah, if you asked me about specific features, and where they're being rolled out, or what's being experimented with, I have no idea that's like, very far away from the things that I deal with on a daily basis. But certainly feature flagging, in general is, is critical. And, you know, not even, not even sort of, for sort of big experimental features, but for almost anything that gets rolled out gets rolled out behind a flag, so it can be turned off. And one I mean, in some ways, that's not, it's not just about sort of experimentation, it's also just a necessity of working at this scale, because the changes to the codebase are coming so quickly, that there's, there's really no possibility of maintaining a long running branch for a new feature. So you can't have this development style where you sort of branch and work on your feature for a long time until it's just perfect. And then like a month and a half later, you merge a big branch back in, there's just no way that would work, it'd be no homerun merges.
Will Vincent 32:41
Yeah, exactly.
Carl Meyer 32:42
So it really forces you into this sort of continuous integration development style, where everything has to land in the same trunk branch, that's constantly being rolled out to production, in fairly small increments. And so that means if you're developing any significantly sized feature change, it forces you to figure out, how can I do this incrementally and backwards compatibly, and have things in a fully working state at every point along the way, and, you know, get the feature off until it's really ready to be rolled out and all that stuff. So So yeah, it's a very big, big part of what we do.
Carlton Gibson 33:20
So you're not using the RM, but you're, like, the issue there is that always happens is migrations, right? It's migrating the database store, and how, when you take the field out, or when you put it in on,
Carl Meyer 33:32
I mean, it's the same issue just even worse, because like,
you know, there's, there's no possibility, you know, within a large number of servers, and obviously, many, many database servers, there's no such thing as like, Okay, I'm gonna synchronize this code rollout with this database migration, everything will roll out together, like every change you make to the data store. I mean, first of all, it's a big deal to roll out any change to the data schema. But any change you make, has to be backwards and forwards compatible, you always have to have things work, if it's new code running against the old schema, or if it's old code, or against the new schema, all that has to work. So there's certain that certainly has a cost in terms of attacks on development speed, but it's just, it's the only option. Can I ask,
Will Vincent 34:23
how long does it take to run the test suite? You know, I mean, cuz I, I know of like small sites that take an hour, you know, cuz they haven't optimized we've done this. And the other thing, I mean, how is it not take months for a little committed Instagram?
Carl Meyer 34:36
So the good news is that we have a lot of tests, pretty good test coverage. The bad news is that I've mentioned earlier about sort of our early Django really pushed you into this path of writing a lot of kind of end to end HTTP tests using the test client. There's a lot of that legacy still in the codebase. So the bad news is that the good news is you have to test the bad news is a lot of them are slow tests. We have a lot of slow tests, you have a very slow test suite. So really, I mean, our our only solution to that is that we throw hardware at it. So when we, I, I literally think that nobody in the world has any idea how long it would take to run the Instagram server test suite, and sequentially, one at a time, because I don't think I think it's been six years since anyone has done that. Like, when we actually run the test suite, it runs sharted out to a bunch of different machines. So like, there's like, a bunch of machines all running tests in parallel, and then sending the results back. And with that kind of a system, you know, we can effectively run the test suite for 1515 or 20 minutes, which is good enough.
Will Vincent 35:44
But yeah, enough to drink, but not, yeah, nothing to drink, but not so much that you're like on Twitter for an hour in between every
Carl Meyer 35:51
Yeah, exactly. And obviously, people don't generally run the whole test suite every time they working locally, like they'll figure out sure which specific tests are relevant, run those locally, and then we'll run the whole test suite before we ship it. So that's
Will Vincent 36:02
something to keep in mind when people tell me about test suites that take more than 20 minutes. Like I heard, Instagram is under 20. So you know, think about I like your point, though, on hardware, though, like cuz that reminds me of the only thing that scale I worked on is Quizlet, which is a top hundred site at the time, and we were on joint and we had the whole executive team come in. And this time, the other thing after a full day, they're like, you just need more hardware. It's just like, the people who wrote it were just like, hardware.
Carl Meyer 36:30
Sometimes throwing computing power at the problem is the most efficient option.
Will Vincent 36:35
Yeah, oh, there's something in a way beautiful. Like, you know, from a business perspective, if you can just throw money and solve an engineering problem. That's sort of great. Actually. It's like,
Unknown Speaker 36:44
you take that is we didn't talk about all but I don't know what that is, or other things that we,
Carl Meyer 36:49
as a company I mentioned that I founded with my siblings in 2008. We are still going strong. I'm not I'm not there anymore on a daily basis. But uh,
Carlton Gibson 36:59
yeah, they made you work on Thanksgiving.
Carl Meyer 37:04
See how things are going? They need help. But yeah, it's still going. I'm still going while still using a lot of Django. I think they're up to six or seven people now. So it's a if you need a Django application or Django site, or application built, and you're looking to contract it out. I mean, ibird does everything from the back end to the UI design and kind of figuring out what the user needs are. So it's an option.
Will Vincent 37:26
Well, maybe one final question. So we're on video. You're not in California, you're remote? I think you've been the whole time you've been promoted Instagram, is that right? That's right. How is that been? Because my sense is most of the team is there. And obviously now everyone's remote, probably curious how, you know, remote learning the last number of years, and then today where everyone's out of the office?
Carl Meyer 37:50
Yeah, I've been remote, pretty much my whole career. So it's kind of like a fish in water. I never really had a big issue with it. Even when I was one of the few people remote. I did travel out to the office in California pretty frequently, which I think is important. There's really no substitute for getting to know people in person at some point. And so just to sort of have those relationships with the team, and I think it was important to make those trips out. But on a daily basis, you know, we have good video conferencing, pretty much everybody's already using, you know, the funny thing is like, the Instagram or Facebook offices are so big that half the time people are meeting over VC anyway, just because they're in different buildings, and they don't want to walk across. That's a really, everybody's already using kind of these remote working tools on a daily basis. Anyway, so I didn't feel that out of place. Using all the same tools. I felt like I was able to keep up. And now of course, yeah, it's the world that everyone lives in. So,
Will Vincent 38:53
right, well, that's one one step further than I remember in. In San Francisco. I was actually I think, on the same block as the Instagram founders, I was on Second Street, you know, around the same time, quit 2010 but it was it you know, it's even if I'm sitting right next to someone, I would send them a, you know, chat message because it's rude to poke someone with headphones on the middle of something, right? It's like, you know, unless it's lunch, like, I don't want to get poked on a question just like, you know, toss it in my chat thing. And I'll say now, here we are, you know, extending that out, but you know, yeah, it makes sense, right? It's, it takes however many fifth 1015 minutes to get into a flow state anyways, and if you're constantly being interrupted, even if you're right there. Yeah, it's rude to poke a coworker. Right. So crossing Can you take us out?
Carlton Gibson 39:40
Well, I can say cause thanks for coming on. It's really interesting chat and love. Let's get it back story. And then, you know, story of Instagram and how they grow up. And you know, that's super, thanks
Carl Meyer 39:52
for Thank you. It's been it's been a pleasure. I've really, really enjoyed the conversation.
Will Vincent 39:55
Well as ever, we are at Django chat.com chat Django on Twitter. Where you guys Say something Carlton or did I talk over you?
Carlton Gibson 40:02
No, it's good. I was just gonna say Join us next time.
Will Vincent 40:04
All right. Bye everyone. Bye Bye. Thanks, guys.