Podcast 062
[Incomplete]
Spolsky: There's a Macintosh on my desktop, that I don't use that often, but when you have a Mac, every day or two, there's this little Apple software updater/globe/thing. It's like this little blue ball on your dock...
Atwood: Uhuh
Spolsky: ...and it just starts jumping up and down. Every couple of days, because it gets excited about something. If you click on it, it tells you that a software update is available.
Atwood:I kinda feel like I wish that a lot of the updates would just silently install.
Spolsky:Yeah
Atwood:Or just go away
Spolsky:It's just not important, they're just not... okay there's some new bug that you guys found in your code.
Atwood:Well it could be important. I mean look at exploits in the browser, thats hugely important.
Spolsky:I know, but there has to be
Atwood:Firefox has had like massive exploits. Like they have like, I think there's bugs in their Javascript engine now, that need to be fixed. That like, if you go to a certain page, you'll literally be owned, that it'll be arbitrary code execution.
Spolsky:But nobody's going to that page! Why would anybody go to that page? You see?
Atwood:OK. So lets say nobody updates their software. So everyone has Firefox 3.5.0...
Spolsky:[Laughs] OK.
Atwood:... and a year from now, there's going to be tonnes of pages that have that exploit on them.
Spolsky: It's true. I guess I'm not arguing that you shouldn't -but there has to be some kind of a cost/benefit analysis
Atwood: It should be silent. I guess what your complaining about is that you have to see it.
Spolsky: But thats because I have to reboot. But thats because I'm going to have to shut all my programs and save all my work, and then reboot, and then re-open all my programs, and find all the work that I was working on, and get back to wherever I was with all of it.
Atwood: Right. Well lets look at Firefox as an example of this, because I think Firefox actually does a good job, but I still have some complaints.
Spolsky: OK
Atwood: In that Firefox does a good job, in that whenever you reset it or it crashes. It remembers like every page you were on, and actually, if you were in the middle of filling out a form, and it's a traditional form - not some crazy Flash thing obviously! It will actually fill in the data that you had there too. So Firefox...
Spolsky: No kidding?
Atwood: ...is a pretty good example of this. It's also an example of the problem, in that every time there's an update to some plug-in, I have to restart. I usually have 20 browser windows open. You gotta realise I have 3 monitors here, so I have just like tonnes of crap on my screen and I'm in the middle of a bunch of crap. So it's still pretty invasive that I have to restart the whole browser just to install, you know, the latest version of Firebug or whatever. So I really think there's 2 things. One is that it should be almost completely transparent, you shouldn't have to think about it, it should just magically work. You just updated, you shouldn't care. You shouldn't care because you never have to see it. The second thing is that it really just needs to be built in, like architectured in. Like, when your starting software?
Spolsky: Right
Atwood: This is one of the great things about web apps right? Because their updated on the server, your always on the latest version and you don't care. And it should be the same principal with desktop software. You should always be on the latest version, because that's just how it works and it's completely transparent.
Spolsky: Well, one thing that always bugged me about Firefox is that, my Firefox window is sitting there all day long, doing absolutely nothing, unless, except for those few minutes of the day when I'm using it; and sometimes I really need to use it, so I launch Firefox, and at that exact moment it says 'This would be the perfect time to do an upgrade!'
Atwood: Yes
Spolsky: So its kind of weird. Like, why is it at startup it does the upgrade? How about at exit or at idle time or something - some time when I'm not using it. It picks the only time when it's annoying, to do the upgrade.
Atwood: Yes. You can be really, really smart about how you do this. Now, the tricky part to me is the architecture of like, an in-place update like that gets really hard. Like I don't know if that's even possible with an executable! It's like you have to restart at some level right? I mean, eugh, it's just a question I guess of just deciding... I don't know, it's tricky.
Spolsky: Depends how you architeture it.
Atwood: That's one of the things about web apps. They sort of have that advantage built-in.
Spolsky: How do they..? No! - But you guys, Stack Overflow, just goes down when you wanna upgrade it, right? How long is it down for?
Atwood: Well it does, it does. But it's a very limited window. Like, maybe like 30 seconds. We could potentially-, we do actually deploy-. Now what I try do on Stack Overflow is: We know when traffic is low because we have Cacti graphs that tell us when traffic is low, um. And we try to the best of our ability to update only during those low traffic periods. We consider any of those low traffic periods fair game. We can deploy as much as we want, [laughs] I mean within reason, not to an absurd degree.
This weekend when Jarod was visiting, he was here from Tuesday to Tuesday. So we were doing lots of stuff on Stack Overflow over the weekend, fixing bugs, adding features, stuff like that. We deployed like, 6 or 7 times over the weekend. But the weekend is like so slow traffic wise.
Spolsky: Couldn't you deploy instantly by just deploying into a new directory and then switching the directory? Just telling IIS to go serve off of this other directory? I've done that with a lot of sites.
Atwood: Your still gonna interrupt...
Spolsky: Nah.
Atwood: Your still gonna interrupt at some level.
Spolsky: Not really, because what happensn is that the old... err... what does IIS do? Does it just cut off people that were getting a page? On the old site I think the people who were getting a page continue to get it, and then the next people to come in...
Atwood: The only people that really get in trouble are the people who are literally clicking the 'Submit' button, like, they've actally submitted new, changed data to the system. For random browsing it's just either a time-out or a big delay, on retreiving the page.
Spolsky: There's definitely ways to build, even though it's more complicated, to build websites so that they have zero outage, even though we're moving to another generation.
Atwood: Right. Well you'd have like another webserver that has the same state...
Spolsky: Yeah, you could have another webserver and just tell your load balancer to start putting the new requests on that one.
Atwood: Yeah, no. You can do it even better than we do. We do it in a very 'ghetto' kinda way.
Spolsky: [laughs]
Atwood: Which is basically, we do the build, which copies all the files... but the advantage is that it's a really narrow window. I mean it's like 30 seconds to a minute at most that there's a potential problem.
[7:00] Fogbugz 7
Spolsky: So. So anyway I was thinking about all these updates. We updated FogBugz. Monday was our official ship date. I'm kinda tired.
Atwood: Fogbugz seven.
Spolsky: There's a Macintosh on my desktop, that I don't use that often, but when you have a Mac, every day or two, there's this little Apple software updater/globe/thing. It's like this little blue ball on your dock...
Atwood: Uhuh
Spolsky: ...and it just starts jumping up and down. Every couple of days, because it gets excited about something. If you click on it, it tells you that a software update is available.
Atwood:I kinda feel like I wish that a lot of the updates would just silently install.
Spolsky:Yeah
Atwood:Or just go away
Spolsky:It's just not important, they're just not... okay there's some new bug that you guys found in your code.
Atwood:Well it could be important. I mean look at exploits in the browser, thats hugely important.
Spolsky:I know, but there has to be
Atwood:Firefox has had like massive exploits. Like they have like, I think there's bugs in their Javascript engine now, that need to be fixed. That like, if you go to a certain page, you'll literally be owned, that it'll be arbitrary code execution.
Spolsky:But nobody's going to that page! Why would anybody go to that page? You see?
Atwood:OK. So lets say nobody updates their software. So everyone has Firefox 3.5.0...
Spolsky:[Laughs] OK.
Atwood:... and a year from now, there's going to be tonnes of pages that have that exploit on them.
Spolsky: It's true. I guess I'm not arguing that you shouldn't -but there has to be some kind of a cost/benefit analysis
Atwood: It should be silent. I guess what your complaining about is that you have to see it.
Spolsky: But thats because I have to reboot. But thats because I'm going to have to shut all my programs and save all my work, and then reboot, and then re-open all my programs, and find all the work that I was working on, and get back to wherever I was with all of it.
Atwood: Right. Well lets look at Firefox as an example of this, because I think Firefox actually does a good job, but I still have some complaints.
Spolsky: OK
Atwood: In that Firefox does a good job, in that whenever you reset it or it crashes. It remembers like every page you were on, and actually, if you were in the middle of filling out a form, and it's a traditional form - not some crazy Flash thing obviously! It will actually fill in the data that you had there too. So Firefox...
Spolsky: No kidding?
Atwood: ...is a pretty good example of this. It's also an example of the problem, in that every time there's an update to some plug-in, I have to restart. I usually have 20 browser windows open. You gotta realise I have 3 monitors here, so I have just like tonnes of crap on my screen and I'm in the middle of a bunch of crap. So it's still pretty invasive that I have to restart the whole browser just to install, you know, the latest version of Firebug or whatever. So I really think there's 2 things. One is that it should be almost completely transparent, you shouldn't have to think about it, it should just magically work. You just updated, you shouldn't care. You shouldn't care because you never have to see it. The second thing is that it really just needs to be built in, like architectured in. Like, when your starting software?
Spolsky: Right
Atwood: This is one of the great things about web apps right? Because their updated on the server, your always on the latest version and you don't care. And it should be the same principal with desktop software. You should always be on the latest version, because that's just how it works and it's completely transparent.
Spolsky: Well, one thing that always bugged me about Firefox is that, my Firefox window is sitting there all day long, doing absolutely nothing, unless, except for those few minutes of the day when I'm using it; and sometimes I really need to use it, so I launch Firefox, and at that exact moment it says 'This would be the perfect time to do an upgrade!'
Atwood: Yes
Spolsky: So its kind of weird. Like, why is it at startup it does the upgrade? How about at exit or at idle time or something - some time when I'm not using it. It picks the only time when it's annoying, to do the upgrade.
Atwood: Yes. You can be really, really smart about how you do this. Now, the tricky part to me is the architecture of like, an in-place update like that gets really hard. Like I don't know if that's even possible with an executable! It's like you have to restart at some level right? I mean, eugh, it's just a question I guess of just deciding... I don't know, it's tricky.
Spolsky: Depends how you architeture it.
Atwood: That's one of the things about web apps. They sort of have that advantage built-in.
Spolsky: How do they..? No! - But you guys, Stack Overflow, just goes down when you wanna upgrade it, right? How long is it down for?
Atwood: Well it does, it does. But it's a very limited window. Like, maybe like 30 seconds. We could potentially-, we do actually deploy-. Now what I try do on Stack Overflow is: We know when traffic is low because we have Cacti graphs that tell us when traffic is low, um. And we try to the best of our ability to update only during those low traffic periods. We consider any of those low traffic periods fair game. We can deploy as much as we want, [laughs] I mean within reason, not to an absurd degree.
This weekend when Jarod was visiting, he was here from Tuesday to Tuesday. So we were doing lots of stuff on Stack Overflow over the weekend, fixing bugs, adding features, stuff like that. We deployed like, 6 or 7 times over the weekend. But the weekend is like so slow traffic wise.
Spolsky: Couldn't you deploy instantly by just deploying into a new directory and then switching the directory? Just telling IIS to go serve off of this other directory? I've done that with a lot of sites.
Atwood: Your still gonna interrupt...
Spolsky: Nah.
Atwood: Your still gonna interrupt at some level.
Spolsky: Not really, because what happensn is that the old... err... what does IIS do? Does it just cut off people that were getting a page? On the old site I think the people who were getting a page continue to get it, and then the next people to come in...
Atwood: The only people that really get in trouble are the people who are literally clicking the 'Submit' button, like, they've actally submitted new, changed data to the system. For random browsing it's just either a time-out or a big delay, on retreiving the page.
Spolsky: There's definitely ways to build, even though it's more complicated, to build websites so that they have zero outage, even though we're moving to another generation.
Atwood: Right. Well you'd have like another webserver that has the same state...
Spolsky: Yeah, you could have another webserver and just tell your load balancer to start putting the new requests on that one.
Atwood: Yeah, no. You can do it even better than we do. We do it in a very 'ghetto' kinda way.
Spolsky: [laughs]
Atwood: Which is basically, we do the build, which copies all the files... but the advantage is that it's a really narrow window. I mean it's like 30 seconds to a minute at most that there's a potential problem.
[7:00] Fogbugz 7
Spolsky: So. So anyway I was thinking about all these updates. We updated FogBugz. Monday was our official ship date. I'm kinda tired.
Atwood: Fogbugz seven.
