Spolsky: Today is the day that we did not launch, although we planned to. But then... We'll wait for another week.Atwood: Yeah, well, the good news on that is that we did actually figure out what that problem was.
Spolsky: Oh, oh, I want to hear, I want to hear, I want to hear.Atwood: Eh. So, it was a third party library. Indirectly, I mean. It's the third party library, and our particular use of it. It was Log4Net.
Spolsky: Oh!Atwood: We were logging in such a way that the log.... during the log call was triggering another log call. Which is normally okay, but with the load that we have, eventually they would happen so close together that there's also a lock. So, there's two locks going on there. There's a lock of like disposing of the database stuff that's going on. Then there's lock of like actually writing to a file...
Spolsky: Hm!Atwood: And... Huh... They happen in the opposite order, so it's like a classic deadlock, right. So, you release the lock on the database, then you release the lock on the file. And then the other call was doing in the other order. And they were happening so fast that... it was deadlocking eventually. And it was one of those things that would happen.. like... it was very [intermittent [??]], right.
Spolsky: Right.Atwood: So we had to dust-off Win Debug.
Spolsky: How on Earth do you find things like that?Atwood: Well. You bust out Win Debug. One nice feature in Windows 2008, and I think this is in Vista as well. In Task Manager, you can right-click a task and take a dump of it.
Spolsky: Yeah!Atwood: Like right there.
Spolsky: Aha.Atwood: So we took a dump of the W3Service process, and...
Spolsky: Ha ha, take a dump.Atwood: Yeah, I know, any time you do this it's like.. It's like the territory for jokes. It's just...
Spolsky: [giggles]Atwood: [laughs]. And then we loaded up a... Win... Debug.
Spolsky: Windbg! YeahAtwood: ...and then some .Net managed extensions you can, sort of load. You need like a chi-chi to figure out what the commands are. And then you load the dump, and you load the manage tools. And then you can sort of just investigate all of the threads. You can take the "Show me all the manage threads." And then say "Show me what's the call stack was for that thread." And what we saw was like tons and tons of threads that were all going "Hey, I would like to log something..." And it was like "Hmmmm... [laughs]... Interesting!" Right, you have like 80 threads that all try to write something to the log. So... Right then we kind of knew where the problem was.
And then somebody on Twitter actually volunteered to help us diagnose the dump. So I put it up on our server, and he a.... he nailed... he had a great description of it, like line by line, blow by blow of exactly what was happening. I mean, I'm... I'm competent enough to sort of figure out roughly what was going on, but he really knew this stuff and really helped us out, and I do appreciate that.
Spolsky: That's really awesome.
Spolsky: I'd never .. no ... I never do... I nev... But you know I don't... I don't think I've ever worked on code that is sort of operational in the same way.
Atwood: ah hm.
Spolsky: Eh.. because we definitely eh.. put a lot more ... oh you know, you know what, I did, at Juno we used to have all kinds of logging.
The trouble is that my philosophy has always been that you .. you.. you have a tendency to wanna log everything. But then you just get logs that are, you know, a hundred megabyte per user and you get thirty of them a minute and it can't possibly be analyzed or stored in any reasonable way. So the next thing you have to do is to start culling your logs or just have different levels of debugging, where it's like in high debug mode everything is logged and in low debug mode nothing is logged. And... it's kind of hard to figure out what you really want in a log. You you know you know .. a lot of logs, like I think of the logging that we did in Juno, where people would call with a complaint and you try to figure out where this program is crashing. And obviously a log of the crash, that's easy. Ehm, but then there's some line above the crash which hopefully gives you a lot of information about where it happened. And there's some line you don't see that should have been after that, after the crash, but it never got there 'cause it crashed sometime before there. And essentially what you're doing as you're adding logging, is you're doing binary search, right, where you're sticking in like "well gosh, I got to here and then got to there. But there's an awful lot of code between point A and point B. So let's make an A you know half-way from A to B, log point of some sort". Then you put that in and then you eliminate 50 percent of the possible places to look for your crash.
Um, but I've never really been able to...
Atwood: I mean that, ironically, to troubleshoot this hang, which turned out to be because of logging, we were adding more logging.
Atwood: The joke just writes itself! The joke just writes itself, right...
Spolsky: It does... How many... How many third-party tools do you have... uhh... How many third-party tools are a part of the StackOverflow code base?
Atwood: Well, okay, so... [chuckles] Uh, Dare [pronounces it as the English word "dare"] Obasanjo [pronounces it "oh-bih-san-ho"]... I don't know if I'm pronouncing it correctly.
Spolsky: Okay, "Dare" [pronounces it "daray"]... Obasanjo [pronounces it "oh-bih-san-ja"]... It's "Dare."
Atwood: Is it "Dare"?
Atwood: Really... Okay, I didn't know that. Well, I've learned something. But he had a whole blog entry about how, you know, I had chosen to write my own sanitizer, and that was a very deliberate choice for me...
Spolsky: Mm hmm.
Atwood: ...for a number of reasons that I won't get into. But he was very critical of this, because, of course there were bugs in the sanitizer...
Spolsky: Mm hmm.
Atwood: ...which there were going to be, and to me, it's about, like, it's about your velocity; it's not about where you are; it's about where you're going, and we're gonna fix that stuff, right, and I'm making the sanitizer public as well, so other people can have a sanitizer that's not ten thousand lines of code, and ridiculous, and uh, so there's a philosophy there of building something that's reusable for everyone. Um, but I thought it was ironic, because he was talking about how developers should just pick a third-party library and go with it, and I think obvio... it's a balancing act, because we picked this logging library, right, which kind of caused a problem for us, right, I mean partially it was the way we were using it, but the way it was locking the files was a design issue in terms of the way Log4Net works.
Atwood: So I... I think it's a trade-off. I don't think it's always as clear-cut as "you should always pick a library" or "you should never pick a library," right? I think there's always some in-between there. So, for us, I'm definitely a minimalist—I don't like third-party libraries; I feel like we have a giant third-party library called "Windows," called ".NET"... huh... ASP.NET MVC is technically a third-party library. Um, but these are, you know, major vendor stacks. And I do feel like—as much as we talk about open source and stuff—there's a certain level of quality you associate with these major first-party stacks, right, whether it's from Apple or Microsoft or Sun or whoever. That may or may not be true, but hopefully usually is true: that these things are really heavily tested.
Spolsky: There is definitely, yeah, there is definitely... I mean, there's something I've learned over the years, and, you know, I started out with working on the Excel team, um... The developers on that team had a motto, which was "Find the dependencies and eliminate them." You know, they had their own compiler; they would not use untested libraries from other groups at Microsoft even...
Atwood: I love that they had their own compiler. That is so hardcore. I can't even, like, I could not even hang out with those guys... right... that hardcore.
Spolsky: Hey, well, we have our own compiler, man.
Spolsky: Let me tell you why they had their own compiler: They had their own compiler because Excel was getting huge, and just compiled 8086 was just too large to fit on floppy disks and to fit in memory. You know, we were really trying to cram things in there. And so they developed a pcode compiler, which basically... you know, it's like bytecode. They called it pcode. This is a very old technique, and it compiled Excel into an imaginary machine, a virtual machine, which was a lot more expressive that an 8086, and had all kinds of additional features, and so the compiled code is about one-third the size, and in a lot of situations this made the performance a lot faster. So, for example, in those days when almost everybody was running programs off of floppy disks, the chances... Or no, not floppy, but the 3.5-inch, not-so-floppy disks. But the read time on those things is really really slow, so if you could launch your app—if your app was smaller at the time that you read it from disk—it didn't matter if it ran a little bit slower. The whole... the overall experience would be a lot faster. So if you could fit in memory without swapping, then obviously the whole thing would run faster, so it was worth doing this pcode thing for a long time, and about the time of Excel 5.0, the bit flipped on that and it suddenly became... suddenly everybody had hard drives, and nobody really cared about the size of the executable, and it was okay to have about a, I think, a four-megabyte executable instead of a one-megabyte executable, and so they got rid of that pcode back-end. But even then I think they had their own compiler for a while because in order to right really really efficient code, they wanted to be able to control... oh this is a long story.
Spolsky: So this was this feature that they eventually got added to the regular Microsoft C compiler called base pointers, and I think then they stopped using their own compiler. But their philosophy was really not to trust anybody and to have control over everything so that there's some hope they can get it to work without having an external dependency. You know, I've sort of taken this with me a long ways, and every time I've failed to do that I've tended to regret it. Every time we've put outside technology into Fog Bugz we've regretted it. There's a lot of these excellent components, and they are really great components, that are made by vendors like .Net components and they're widgets like the cool calendar widget dropdown that you put into your web page and all that kind of stuff. And inevitably what I've found is that they are good enough for enterprise code, like internal apps that you're using at the insurance company, and they're just never good enough for the kind of app you want to ship that has to be perfect. Somehow there is something that's not commercial quality about them. You know it's fine if there's 20 people using it, and they're all using it the same way, it allows you to in fifteen seconds put a calendar dropdown into something. But then you'll get to some customer who says, "You know we don't start our week's on Sunday in my country". And you'll say, "Oh", and you'll find out this library doesn't have that feature. As a hypothetical example.
Spolsky: Which I ...
Atwood: I believe that's one advantage of some of the web stuff is because everything is just public facing by default. You don't have sort of this, this internal development ghetto effect. Like... 'cuz I... we have... I mean, to be fair, any talk of dependencies, we have tons of dependencies, right? It's just a question of what dependencies you wanna take, like, I mean, jQuerry is a dependency, right?
Atwood: We're using the WMD control, that's a dependency.
Atwood: There's these little add-ons for jQuery that...
Spolsky: But you know what, if there was a bug in... if you found a bug in jQuery, you would just go, edit the source and you would be shipping your own private version of jQuerry and problem solved. And it wouldn't be ideal, but at least you wouldn't be screwed.
Atwood: Right ... We have actually done that and let me actually give an example. So the WMD editor has a bug with international keyboards. Like there's no way we would have found this because we don't use international keyboards but obviously some of the people that use StackOverflow do and
Atwood: They were, I think, understandably very annoyed because they couldn't enter like right bracket. Which is like an important key particularly in markdown. That's one of the key, like one of the delimiters you use sometimes.
Spolsky: Yeah, the...
[To be transcribed]
Atwood: But people who do something like that, they don't want attention or anything, they're not feeding on it at all, they're not interested in things like attention at all. The work is it's own reward, it doesn't matter if anyone is looking. Have you seen that thing on the ... I meant to blog about this but the whole concept of just, not looking at things, to basically discourage them. Or that, conversely, looking at things encourages them, like the whole Paris Hilton thing and just talking about these things over and over incessantly actually reinforces the whole trend. There was a series of children's books I don't know if you've heard them, their called: "The Great Brain". It's set in Utah -
Atwood: - at the turn of the century. I got these books as a kid and I was totally obsessed with them, because the Great Brain is all about a family. I don't remember the family's name, but there is one central character J.D. who is the Great Brain. Essentially he's always thinking up ways to, essentially social engineering before we had that word in computer circles. Basically getting people to do what you want them to do, completely of their own volition. The Great Brain is basically this genius of a kid, who is using all these social engineering exploits to get away with all this crazy stuff. In that family, if they found out the Great Brain was doing this stuff then of course he would get punished, but the ultimate penalty was what they called the silent treatment. The silent treatment meant that nobody would talk to you or acknowledge you for a certain period of time. They would give you food and stuff, but they wouldn't talk at you. It was just stunning in the book, you don't really think about this stuff as a kid (I was like 10 or so), how desperate it is as a person, as a social being, when nobody will acknowledge you. How profoundly affecting that is, right? Even the Great Brain, as a smart kid, hated the silent treatment and would do anything he could to avoid getting the silent treatment because it was just such a brutal penalty. I remember Jason Kottke talking about an episode of The Simpsons where these animated statues came to life, and the way they got rid of them was they started chanting "Just Don't Look!"
Atwood:I also try to copy a lot of things I've seen online that have been successful, like conventions. Let me give you a specific example: PHP-BB, and I'm sure there are other web discussion boards that do this too but PHP-BB is the one I know, has this editing convention. When you post you can edit your own posts, and I noticed that when people are using PHP-BB that right after you post something you'll always notice some goofy mistake that you made, like immediately. This happens to me 9 times out of ten I'll post and think "oh, I should have talked about this" or "I missed that word," so you immediately go in and edit. At a certain threshold these are not treated as real edits, they're treated as just going back in time to pretend that it is the post you originally made. It doesn't kick off the whole auditing trail of you having edited it 50 times. One of the first things we did in stackoverflow is actually implement that. I remember talking to Jeff Dalgas (sp?) about that, and he's like "why do we have to have this?" I said, "you don't understand, this feature has to be in there on day 1 otherwise we're going to have so many revisions that are just in the first minute or two after posting and are just silly little things that are being corrected."
Spolsky: It's about a one year project. We've got the same film maker as we had last time, except I'm basically interviewing everybody at Fog Creek. You know, three or four times, to get footage for this thing. And then we'll splice it all together, and the current plan is to have several different formats of this movie. There'll be a lot of short form little piece, like four minutes, we can put up on YouTube. If you want to hear about source code control, or you want to learn about hiring programmers or interviewing programmers or the phone interview. They'll be these little pieces that you can just download freely on the internet or watch. And then there's a [sort of a] more substantial ninety minute version, which is probably too much to distribute over the internet. But it'll probably distribute in DVD form. Maybe by that time, there'll be a good way to distribute a [you know] ninety minute DVD length thing over the internet. At kind of the high end, and what will hopefully pay for this whole operation, will be a corporate training video. Maybe a five to six hour thing that a team might buy inside a corporation to learn how to do software development a little bit better, and it'll just go into much more depth. There'll be like a whole hour of training on how to conduct interviews for programmers and a whole hour on how to setup your tools and that kind of stuff.
Atwood: Two things; we have a wiki, for people who can't listen to this. Where people can contribute transcriptions of our incredibly boring podcasts, and we thank you very much for that. Although I do have one request for the transcriptionists, and the ironic thing is, you're going to transcribe this, which I think is hilarious. When you transcribe, don't write down every time I say 'uh' or pause or 'yeah'. Make me sound awesome, that's my one request for the transcriptionists.
Spolsky: It doesn't have to be word for word. It doesn't necessarily read as well when it's word for word you can leave [out] 'uhms' and 'uhs'.
Atwood: In fact, leave out whole. If you think it reads better a certain way, just make me say whatever makes the transcriptionists sound the most awesome.
Spolsky: And it's wiki, go ahead and edit it.
Atwood: People edit anyway, you're right, it's hilarious. I've been reading the [revisions], it's very funny.
Atwood: The other thing is, if you do contribute to the wiki. Since our beta has been pushed back a week, this will get you in the same day, to the StackOverflow beta. If you want to be in, just email me after you've done a little bit of transcription, one minute or whatever you're comfortable with. If you want to get your question answered on the air, send a less than 90 second recording to email@example.com, we will put it in the queue and hopefully answer it on the next podcast.
Spolsky: Alright, that's it. Thank you very much, see you next week!
Atwood: See you next week.