Spolsky: Today is the day that we did not launch, although we planned to. But then... We'll wait for another week.
Spolsky: Oh, oh, I want to hear, I want to hear, I want to hear.
Spolsky: How on Earth do you find things like that?
Spolsky: [Then [??]] take a dump.
Spolsky: Windbg! Yeah
And then somebody on Twitter actually volunteered to help us diagnose the dump. So I put it up on our server, and he a.... he nailed... he had a great description of it, like line by line, blow by blow of exactly what was happening. I mean, I'm... I'm competent enough to sort of figure out roughly what was going on, but he really knew this stuff and really helped us out, and I do appreciate that.
Spolsky: That's really awesome.
Spolsky: I'd never .. no ... I never do... I nev... But you know I don't... I don't think I've ever worked on code that is sort of [operational] in the same way.
Atwood: ah hm.
Spolsky: Eh.. because we definitly eh.. put a lot more ... oh [you know] I did, at Yuno we used to have all kinds of logging.
The trouble is that my philosophy has always been that you .. you.. you have a tendency to wanna log everything. But then you just get logs that are, you know, a hundred megabyte per user and you get thirty of them a minute and it can't possibly be analyzed or stored in any reasonable way. So the next thing you have to do is to start culling your logs or just have different levels of debugging, where it's like in high debug mode everything is logged and in low debug mode nothing is logged. And... it's kind of hard to figure out what you really want in a log. You you know you know .. a lot of logs, like I think of the logging that we did in Yuno, where people would call with a complaint and you try to figure out where this programm is crashing. And obviously a log of the crash, that's easy. Ehm, but then there's some line above the crash which hopefully gives you a lot of information about where it happend. And there's some line you don't see that should have been after that, after the crash, but it never got there 'cause it crashed sometime before there. And essentially what you're doing as you're adding logging, is you're doing binary search, right, where you're [sticking in] like "well gosh, I got to here and then got to there. But there's an awful lot of code between point A and point B. So let's make an A you know half-way from A to B, log point of some sort". Then you put that in and then you eliminate 50 percent of the possible places to look for your crash.
Um, but I've never really been able to...
Atwood: I mean that, ironically, to troubleshoot this hang, which turned out to be because of logging, we were adding more logging.
Atwood: The joke just writes itself! The joke just writes itself, right...
Spolsky: It does... How many... How many third-party tools do you have... uhh... How many third-party tools are a part of the StackOverflow code base?
Atwood: Well, okay, so... [chuckles] Uh, Dare [pronounces it as the English word "dare"] Obasanjo [pronounces it "oh-bih-san-ho"]... I don't know if I'm pronouncing it correctly.
Spolsky: Okay, "Dare" [pronounces it "daray"]... Obasanjo [pronounces it "oh-bih-san-ja"]... It's "Dare."
Atwood: Is it "Dare"?
Atwood: Really... Okay, I didn't know that. Well, I've learned something. But he had a whole blog entry about how, you know, I had chosen to write my own sanitizer, and that was a very deliberate choice for me...
Spolsky: Mm hmm.
Atwood: ...for a number of reasons that I won't get into. But he was very critical of this, because, of course there were bugs in the sanitizer...
Spolsky: Mm hmm.
Atwood: ...which there were going to be, and to me, it's about, like, it's about your velocity; it's not about where you are; it's about where you're going, and we're gonna fix that stuff, right, and I'm making the sanitizer public as well, so other people can have a sanitizer that's not ten thousand lines of code, and ridiculous, and uh, so there's a philosophy there of building something that's reusable for everyone. Um, but I thought it was ironic, because he was talking about how developers should just pick a third-party library and go with it, and I think obvio... it's a balancing act, because we picked this logging library, right, which kind of caused a problem for us, right, I mean partially it was the way we were using it, but the way it was logging the files was a design issue in terms of the way it logged for networks.
Spolsky: edit me!
Atwood: edit me!