Spolsky: Today is the day that we did not launch, although we planned to. But then... We'll wait for another week.
Spolsky: Oh, oh, I want to hear, I want to hear, I want to hear.
Spolsky: How on Earth do you find things like that?
Spolsky: [Then [??]] take a dump.
Spolsky: Windbg! Yeah
And then somebody on Twitter actually volunteered to help us diagnose the dump. So I put it up on our server, and he a.... he nailed... he had a great description of it, like line by line, blow by blow of exactly what was happening. I mean, I'm... I'm competent enough to sort of figure out roughly what was going on, but he really knew this stuff and really helped us out, and I do appreciate that.
Spolsky: That's really awesome.
Spolsky: I'd never .. no ... I never do... I nev... But you know I don't... I don't think I've ever worked on code that is sort of [operational] in the same way.
Atwood: ah hm.
Spolsky: Eh.. because we definitly eh.. put a lot more ... oh [you know] I did, at Yuno we used to have all kinds of logging.
The trouble is that my philosophy has always been that you .. you.. you have a tendency to wanna log everything. But then you just get logs that are, you know, a hundred megabyte per user and you get thirty of them a minute and it can't possibly be analyzed or stored in any reasonable way. So the next thing you have to do is to start culling your logs or just have different levels of debugging, where it's like in high debug mode everything is logged and in low debug mode nothing is logged. And... it's kind of hard to figure out what you really want in a log. You you know you know .. a lot of logs, like I think of the logging that we did in Yuno, where people would call with a complaint and you try to figure out where this programm is crashing. And obviously a log of the crash, that's easy. Ehm, but then there's some line above the crash which hopefully gives you a lot of information about where it happend. And there's some line you don't see that should have been after that, after the crash, but it never got there 'cause it crashed sometime before there. And essentially what you're doing as you're adding logging, is you're doing binary search, right, where you're [sticking in] like "well gosh, I got to here and then got to there. But there's an awful lot of code between point A and point B. So let's make an A you know half-way from A to B, log point of some sort". Then you put that in and then you eliminate 50 percent of the possible places to look for your crash.
Atwood: Ironically, to troubleshoot this hang, which turned out to be because of logging, we were adding more logging [laughs] The joke just writes itself!
Spolsky: [Laughs] It does. How many third party tools do you have? How many third party tools are part of teh Stack Overflow codebase?
Atwood: Well ok, so [Laughs] Dare Obosanjo He had a whole blog entry about how I'd chosen to write my own sanitiser