Podcast 020
Ads, intro
[1:00]
Spolsky: Today is the day that we did not launch, although we planned to. But then... We'll wait for another week.
Atwood: Yeah, well, the good news on that is that we did actually figure out what that problem was.Spolsky: Oh, oh, I want to hear, I want to hear, I want to hear.
Atwood: Eh. So, it was a third party library. Indirectly, I mean. It's the third party library, and our particular use of it. It was Log4Net.Spolsky: Oh!
Atwood: We were logging in such a way that the log.... during the log call was triggering another log call. Which is normally okay, but with the load that we have, eventually they would happen so close together that there's also a lock. So, there's two locks going on there. There's a lock of like disposing of the database stuff that's going on. Then there's lock of like actually writing to a file...Spolsky: Hm!
Atwood: And... Huh... They happen in the opposite order, so it's like a classic deadlock, right. So, you release the lock on the database, then you release the lock on the file. And then the other call was doing in the other order. And they were happening so fast that... it was deadlocking eventually. And it was one of those things that would happen.. like... it was very [intermittent [??]], right.Spolsky: Right.
Atwood: So we had to dust-off Win Debug.Spolsky: How on Earth do you find things like that?
Atwood: Well. You bust that Win Debug. One nice feature in Windows 2008, and I think this is in Vista as well. In Task Manager, you can right-click a task and take a dump of it.Spolsky: Yeah!
Atwood: Like right there.Spolsky: Aha.
Atwood: So we took a dump of the W3Service process, and...Spolsky: [Then [??]] take a dump.
Atwood: Yeah, I know, any time you do this it's like.. It's like the territory for jokes. It's just...Spolsky: [giggles]
Atwood: [laughs]. And then we loaded up a... Win... Debug.Spolsky: Windbg! Yeah
Atwood: ...and then some .Net managed extensions you can, sort of load. You need like a chi-chi to figure out what the commands are. And then you load the dump, and you load the manage tools. And then you can sort of just investigate all of the threads. You can take the "Show me all the manage threads." And then say "Show me what's the call stack was for that thread." And what we saw was like tons and tons of threads that were all going "Hey, I would like to log something..." And it was like "Hmmmm... [laughs]... Interesting!" Right, you have like 80 threads that all try to write something to the log. So... Right then we kind of knew where the problem was.And then somebody on Twitter actually volunteered to help us diagnose the dump. So I put it up on our server, and he a.... he nailed... he had a great description of it, like line by line, blow by blow of exactly what was happening. I mean, I'm... I'm competent enough to sort of figure out roughly what was going on, but he really knew this stuff and really helped us out, and I do appreciate that.
[3:33]
Spolsky: That's really awesome.
Spolsky: I'd never .. no ... I never do... I nev... But you know I don't... I don't think I've ever worked on code that is sort of [operational] in the same way.
Atwood: ah hm.
Spolsky: Eh.. because we definitely eh.. put a lot more ... oh [you know] I did, at Yuno we used to have all kinds of logging.
The trouble is that my philosophy has always been that you .. you.. you have a tendency to wanna log everything. But then you just get logs that are, you know, a hundred megabyte per user and you get thirty of them a minute and it can't possibly be analyzed or stored in any reasonable way. So the next thing you have to do is to start culling your logs or just have different levels of debugging, where it's like in high debug mode everything is logged and in low debug mode nothing is logged. And... it's kind of hard to figure out what you really want in a log. You you know you know .. a lot of logs, like I think of the logging that we did in Yuno, where people would call with a complaint and you try to figure out where this program is crashing. And obviously a log of the crash, that's easy. Ehm, but then there's some line above the crash which hopefully gives you a lot of information about where it happened. And there's some line you don't see that should have been after that, after the crash, but it never got there 'cause it crashed sometime before there. And essentially what you're doing as you're adding logging, is you're doing binary search, right, where you're [sticking in] like "well gosh, I got to here and then got to there. But there's an awful lot of code between point A and point B. So let's make an A you know half-way from A to B, log point of some sort". Then you put that in and then you eliminate 50 percent of the possible places to look for your crash.
Um, but I've never really been able to...
Atwood: I mean that, ironically, to troubleshoot this hang, which turned out to be because of logging, we were adding more logging.
Spolsky: [laughs]
Atwood: The joke just writes itself! The joke just writes itself, right...
Spolsky: It does... How many... How many third-party tools do you have... uhh... How many third-party tools are a part of the StackOverflow code base?
Atwood: Well, okay, so... [chuckles] Uh, Dare [pronounces it as the English word "dare"] Obasanjo [pronounces it "oh-bih-san-ho"]... I don't know if I'm pronouncing it correctly.
Spolsky: Okay, "Dare" [pronounces it "daray"]... Obasanjo [pronounces it "oh-bih-san-ja"]... It's "Dare."
Atwood: Is it "Dare"?
Spolsky: Yep.
Atwood: Really... Okay, I didn't know that. Well, I've learned something. But he had a whole blog entry about how, you know, I had chosen to write my own sanitizer, and that was a very deliberate choice for me...
Spolsky: Mm hmm.
Atwood: ...for a number of reasons that I won't get into. But he was very critical of this, because, of course there were bugs in the sanitizer...
Spolsky: Mm hmm.
Atwood: ...which there were going to be, and to me, it's about, like, it's about your velocity; it's not about where you are; it's about where you're going, and we're gonna fix that stuff, right, and I'm making the sanitizer public as well, so other people can have a sanitizer that's not ten thousand lines of code, and ridiculous, and uh, so there's a philosophy there of building something that's reusable for everyone. Um, but I thought it was ironic, because he was talking about how developers should just pick a third-party library and go with it, and I think obvio... it's a balancing act, because we picked this logging library, right, which kind of caused a problem for us, right, I mean partially it was the way we were using it, but the way it was logging the files was a design issue in terms of the way it logged for networks.
Spolsky: Right.
Atwood: So I... I think it's a trade-off. I don't think it's always as clear-cut as "you should always pick a library" or "you should never pick a library," right? I think there's always some in-between there. So, for us, I'm definitely a minimalist—I don't like third-party libraries; I feel like we have a giant third-party library called "Windows," called ".NET"... huh... ASP.NET MVC is technically a third-party library. Um, but these are, you know, major vendor stacks. And I do feel like—as much as we talk about open source and stuff—there's a certain level of quality you associate with these major first-party stacks, right, whether it's from Apple or Microsoft or Sun or whoever. That may or may not be true, but hopefully usually is true: that these things are really heavily tested.
Spolsky: There is definitely, yeah, there is definitely... I mean, there's something I've learned over the years, and, you know, I started out with working on the Excel team, um... The developers on that team had a motto, which was "Find the dependencies and eliminate them." You know, they had their own compiler; they would not use untested libraries from other groups at Microsoft even...
Atwood: I love that they had their own compiler. That is so hardcore. I can't even, like, I could not even hang out with those guys... right... that hardcore.
Spolsky: Hey, well, we have our own compiler, man.
Atwood: Yeah...
Spolsky: Let me tell you why they had their own compiler: They had their own compiler because Excel was getting huge, and just compiled 8086 was just too large to fit on floppy disks and to fit in memory. You know, we were really trying to cram things in there. And so they developed a pcode compiler, which basically... you know, it's like bytecode. They called it pcode. This is a very old technique, and it compiled Excel into an imaginary machine, a virtual machine, which was a lot more expressive that an 8086, and had all kinds of additional features, and so the compiled code is about one-third the size, and in a lot of situations this made the performance a lot faster. So, for example, in those days when almost everybody was running programs off of floppy disks, the chance is... Or no, not floppy, but the 3.5-inch, not-so-floppy disks. But the read time on those things is really really slow, so if you could launch your app—if your app was smaller at the time that you read it from disk—it didn't matter if it ran a little bit slower. The whole... the overall experience would be a lot faster. So if you could fit in memory without swapping, then obviously the whole thing would run faster, so it was worth doing this pcode thing for a long time, and about the time of Excel 5.0, the bit flipped on that and it suddenly became... suddenly everybody had hard drives, and nobody really cared about the size of the executable, and it was okay to have about a, I think, a four-megabyte executable instead of a one-megabyte executable, and so they got rid of that pcode back-end. But even then I think they had their own compiler for a while because in order to right really really efficient code, they wanted to be able to control... Now this is a long story.
[9:39]
Spolsky: but a pointer on an 80386... for a while the 80386 was the target. On an 80386, a pointer consisted of (or even on the 8086 in general) a pointer consisted of two parts, the segment and the offset. So it's like "where do you want to start your pointer?" and then "what's your offset inside there?". And you couldn't just indirectly say "here's my pointer, just do something with this"
Atwood: edit me!
Spolsky: edit me!
[To be transcribed]
[20:26]
Atwood: But people who do something like that, they don't want attention or anything, they're not feeding on it at all, they're not interested in things like attention at all. The work is it's own reward, it doesn't matter if anyone is looking. Have you seen that thing on the ... I meant to blog about this but the whole concept of just, not looking at things, to basically discourage them. Or that, conversely, looking at things encourages them, like the whole Paris Hilton thing and just talking about these things over and over incessantly actually reinforces the whole trend. There was a series of children's books I don't know if you've heard them, their called: "The Great Brain". It's set in Utah -
Spolsky: Yeah
Atwood: edit me!
Spolsky: edit me!
[To be transcribed]
[20:26]
Atwood: But people who do something like that, they don't want attention or anything, they're not feeding on it at all, they're not interested in things like attention at all. The work is it's own reward, it doesn't matter if anyone is looking. Have you seen that thing on the ... I meant to blog about this but the whole concept of just, not looking at things, to basically discourage them. Or that, conversely, looking at things encourages them, like the whole Paris Hilton thing and just talking about these things over and over incessantly actually reinforces the whole trend. There was a series of children's books I don't know if you've heard them, their called: "The Great Brain". It's set in Utah -
Spolsky: Yeah
Atwood: - at the turn of the century. I got these books as a kid and I was totally obsessed with them, because the Great Brain is all about a family. I don't remember the family's name, but there is one central character J.D. who is the Great Brain. Essentially he's always thinking up ways to, essentially social engineering before we had that word in computer circles. Basically getting people to do what you want them to do, completely of their own volition. The Great Brain is basically this genius of a kid, who is using all these social engineering exploits to get away with all this crazy stuff. In that family, if they found out the Great Brain was doing this stuff then of course he would get punished, but the ultimate penalty was what they called the silent treatment. The silent treatment meant that nobody would talk to you or acknowledge you for a certain period of time. They would give you food and stuff, but they wouldn't talk at you. It was just stunning in the book, you don't really think about this stuff as a kid (I was like 10 or so), how desperate it is as a person, as a social being, when nobody will acknowledge you. How profoundly affecting that is, right? Even the Great Brain, as a smart kid, hated the silent treatment and would do anything he could to avoid getting the silent treatment because it was just such a brutal penalty. I remember Jason Kottke talking about an episode of The Simpsons where these animated statues came to life, and the way they got rid of them was they started chanting "Just Don't Look!"
Spolsky: (laughs)
Atwood: This is led by Lisa Simpson, who says "Just don't look at them and they'll go away!"
Spolsky: (laughs)
Atwood: Its amazing how powerful that philosophy is. If there are things happening you don't like just don't talk about it or don't give it any attention, and look at the things that you actually care about and actually want (to have happen).
Spolsky: Thats right. That was like the Clinton administration's policy on Rwanda.
Atwood: (uncertain chuckle) Well thats the issue of social injustices, which I think is a little bit different.
Spolsky: There are definitely people who are attention seeking, but you know what? That the thing about trolls, "don't feed the trolls" or whatever. You know what, trolls are doing a great service to the Internet, they're making it entertaining and interesting.
Atwood: Absolutely. There's definitely been, and I think I referred to it in the previous podcast, "youthful experimentation" at stackoverflow. We're still in fact seeing that.
Spolsky: (chuckles)
Atwood: We're still seeing people just try stuff to push buttons and see what they can get away with.
Spolsky: One of the things that will happen, and it happened with IRC all the time, when people get addicted to a social technology or something. They're addicted to the site, the love it, they come and they answer all the questions, but at some point there is just not enough of the main content that everybody is enjoying for them to entertain themselves. They have to find ways to entertain themselves for the next three hours after they've spent the 8 hours doing the regular thing they were supposed to be doing. So on IRC they would start these little flame-bot wars and they would write little bots to protect themselves against the wars, and they would try to cause splits so that they could take over somebody's nickname, and that kind of stuff. They weren't doing the kind of stuff that IRC was for, which was chatting, they'd taken it to the next level because they ran out of people to chat with and that became boring. So they started kind of attacking the system itself.
Atwood: Right, and we're definitely looking at that. I mentioned that CAPTCHA thing, and thats the next piece of the puzzle that has to go in. We have rate limiting mechanisms. One of the early things that happened to us was -
Spolsky: Flooding.
Atwood: - somebody wrote a bot that would just revise posts every minute to keep it on the top of the stack. Actually there is a certain amount of people doing that still, which I'm trying to discourage. They way I like to discourage things is, where possible, creating rules in the system that make that behavior not desirable. Not negative, necessarily, but things happen that make it not worth much to you. So let me give you a specific example in that scenario. You have the user who's just editing their own posts every three hours so that its always on the top of the stack. We have this concept now, its actually implemented though I talked about it in previous podcasts, of the community owned post. Because one of the great divides in stackoverflow is that we have this ownership system where you get voted up and down, your content gets voted up and down, and that affects your reputation. You own stuff, so when you post something you own it. Then you contrast that with the Wikipedia model which is that nobody seems to own it, and we're trying to do both of those things. At the transition point we came up with a couple rules. The initial rule I had was that edits by four different people will cause a post to switch from being owned by Joel, for example, to being owned by the community user. At that point you don't lose any reputation that you got up to that point, but any future upvotes on that content don't go to anybody they go to the content. I think this is the way it should be. Ultimately you're voting on the content more than the person anyway, so hopefully people are ok with this. Seeing that people kept editing stuff over and over I bent the rules a little bit and said ok, if you edit your own thing more than N times then it also becomes a community owned post. There is no real value to the user, in terms of getting additional reputation, to bump stuff up to the top of the stack anymore because if you edit your own thing enough you won't get any reputation from it. It behooves you to only edit it once, or however many times you need to edit it but hopefully no more than once, and just let it sit there and have people find it organically and naturally the way it is supposed to happen.
Spolsky: That will happen more when we're open to Google for searches. I think right now one of the problems with stackoverflow that some people have been experiencing is that they ask a question, and its a little too esoteric to get a response right away, and then it disappears from view for a while. Once the site has a much larger critical mass of people and is searchable by Google, those questions will naturally have people come to them so they won't need to try these little tricks to get them in front of people again to get an answer.
Atwood:
I also try to copy a lot of things I've seen online that have been successful, like conventions. Let me give you a specific example: PHP-BB, and I'm sure there are other web discussion boards that do this too but PHP-BB is the one I know, has this editing convention. When you post you can edit your own posts, and I noticed that when people are using PHP-BB that right after you post something you'll always notice some goofy mistake that you made, like immediately. This happens to me 9 times out of ten I'll post and think "oh, I should have talked about this" or "I missed that word," so you immediately go in and edit. At a certain threshold these are not treated as real edits, they're treated as just going back in time to pretend that it is the post you originally made. It doesn't kick off the whole auditing trail of you having edited it 50 times. One of the first things we did in stackoverflow is actually implement that. I remember talking to Jeff Dalgas (sp?) about that, and he's like "why do we have to have this?" I said, "you don't understand, this feature has to be in there on day 1 otherwise we're going to have so many revisions that are just in the first minute or two after posting and are just silly little things that are being corrected."[28:20]
[57:56]
Spolsky: [We will actually have] I'm in the process of making another movie. Did we talking about this on the podcast yet?
Atwood: Oh really? You sent me a copy of Aardvark'd. I have seen that.
Spolsky: That was our first movie. That was like a slightly serious reality TV show about the interns building Copilot. It was probably better than what on MTV. But the truth is the audience, the Joel on Software audience, really wanted something more technical and a lot of people wrote to me and said "You know, I really wanted to learn how software is developed at Fog Creek", and this movie was just a little bit too light on that content. This is going to be much more hardcore. SO it's really going to be about how software is developed at Fog Creek.
Atwood: Ooh wow... Has this started?
Spolsky: It's about a one year project. We've got the same film maker as we had last time, except I'm basically interviewing everybody at Fog Creek. You know, three or four times, to get footage for this thing. And then we'll splice it all together, and the current plan is to have several different formats of this movie. There'll be a lot of short form little piece, like four minutes, we can put up on YouTube. If you want to hear about source code control, or you want to learn about hiring programmers or interviewing programmers or the phone interview. They'll be these little pieces that you can just download freely on the internet or watch. And then there's a [sort of a] more substantial ninety minute version, which is probably too much to distribute over the internet. But it'll probably distribute in DVD form. Maybe by that time, there'll be a good way to distribute a [you know] ninety minute DVD length thing over the internet. At kind of the high end, and what will hopefully pay for this whole operation, will be a corporate training video. Maybe a five to six hour thing that a team might buy inside a corporation to learn how to do software development a little bit better, and it'll just go into much more depth. There'll be like a whole hour of training on how to conduct interviews for programmers and a whole hour on how to setup your tools and that kind of stuff.
Atwood: Wow. So you're like Howard Stern. You're like the kind of the media now. You have like your movie. You've got your podcasts with me. You've got, you know, your blog, your not-a-blog.
Spolsky: But I mean, you know, that was the original idea behind this podcast remember how I said, like, "There's some people that you can reach in certain channels, and they need to hear you. I don't listen to Eminem's music, blog I saw the movie 8-mile, or miles... whatever it was. Different people, you will reach in different channels. Part of the goal is to do that, and the other part of the goal is: I have an inherent inability to ever give up. And instead of just saying, "You know, we tried to make a movie, and it wasn't as good as it should have been, and so maybe we're not so good at making movies.", we just said "You know, let's learn from the experience. Let's try and make a two point oh, that's just much better."
Atwood: I didn't really hear a lot of negative stuff about it.
Spolsky: You know who liked it? Programmer's girlfriend, wives and families. Because there's nothing about it that's inaccessible to somebody who knows nothing about programming. And it really does convey to the normal person a lot about what a programmer's life is like. Just the sort of feel or being a programmer. The fact that programmers get into these conversations about things. You know that was probably, I thought, one of the highlights of the movie - when we told the interns that their job was to figure out if they could jump across to the ledge of the neighboring building if there was a fire. Could they jump out the window and make it on to the ledge of the neighboring building? And programmers love to have conversations like that, and they immediately leapt up to the white board and started drawing equations from physics and stuff like that, and they tried to work it out. And that's really what our conversations are like, what our lives are like. And to geeks, nobody knows--, if you're a geek you probably don't think anything is strange about this. But to a normal human being... this is very bizarre. That we talk that way, and that we think about these kind of things, and that we then need to solve the problem mathematically, to decide if you can make it to the neighboring ledge before you died.
Atwood: Right... Well that's cool. I'm looking forward to that coming out. Looks like it'll be a while. I definitely recommend people who haven't seen it, check out Aardvark'd. I enjoyed it. I didn't think it lacked anything.
Spolsky: The way to find that is ProjectAardvark.com, and there's link to the movie. ProjectAardvark.com is a blog that the interns kept that summer. It was a couple of years ago.
[1:02:14]
[1:02:50]
Atwood: Two things; we have a wiki, for people who can't listen to this. Where people can contribute transcriptions of our incredibly boring podcasts, and we thank you very much for that. Although I do have one request for the transcriptionists, and the ironic thing is, you're going to transcribe this, which I think is hilarious. When you transcribe, don't write down every time I say 'uh' or pause or 'yeah'. Make me sound awesome, that's my one request for the transcriptionists.
Spolsky: It doesn't have to be word for word. It doesn't necessarily read as well when it's word for word you can leave [out] 'uhms' and 'uhs'.
Atwood: In fact, leave out whole. If you think it reads better a certain way, just make me say whatever makes the transcriptionists sound the most awesome.
Spolsky: And it's wiki, go ahead and edit it.
Atwood: People edit anyway, you're right, it's hilarious. I've been reading the [revisions], it's very funny.
Atwood: The other thing is, if you do contribute to the wiki. Since our beta has been pushed back a week, this will get you in the same day, to the StackOverflow beta. If you want to be in, just email me after you've done a little bit of transcription, one minute or whatever you're comfortable with. If you want to get your question answered on the air, send a less than 90 second recording to podcast@stackoverflow.com, we will put it in the queue and hopefully answer it on the next podcast.
Spolsky: Alright, that's it. Thank you very much, see you next week!
Atwood: See you next week.
[1:04:04]
[Outro]
