Revision Diff: Podcast 020

Podcast 020 Revision #23, 9/4/2008 2:57 AM User: "22:38 to 23:21" Tags: (None) Previous Next	Podcast 020 Revision #24, 9/4/2008 3:01 AM 60.242.207.109: "Added [57:56] to [1:02:14] + misc spelling corrections" Tags: (None) Previous Next
Ads, intro [1:00] Spolsky: Today is the day that we did not launch, although we planned to. But then... We'll wait for another week. Atwood: Yeah, well, the good news on that is that we did actually figure out what that problem was. Spolsky: Oh, oh, I want to hear, I want to hear, I want to hear. Atwood: Eh. So, it was a third party library. Indirectly, I mean. It's the third party library, and our particular use of it. It was Log4Net. Spolsky: Oh! Atwood: We were logging in such a way that the log.... during the log call was triggering another log call. Which is normally okay, but with the load that we have, eventually they would happen so close together that there's also a lock. So, there's two locks going on there. There's a lock of like disposing of the database stuff that's going on. Then there's lock of like actually writing to a file... Spolsky: Hm! Atwood: And... Huh... They happen in the opposite order, so it's like a classic deadlock, right. So, you release the lock on the database, then you release the lock on the file. And then the other call was doing in the other order. And they were happenning so fast that... it was deadlocking eventually. And it was one of those things that would happen.. like... it was very [intermittent [??]], right. Spolsky: Right. Atwood: So we had to dust-off Win Debug. Spolsky: How on Earth do you find things like that? Atwood: Well. You bust that Win Debug. One nice feature in Windows 2008, and I think this is in Vista as well. In Task Manager, you can right-click a task and take a dump of it. Spolsky: Yeah! Atwood: Like right there. Spolsky: Aha. Atwood: So we took a dump of the W3Service process, and... Spolsky: [Then [??]] take a dump. Atwood: Yeah, I know, any time you do this it's like.. It's like the territory for jokes. It's just... Spolsky: [giggles] Atwood: [laughs]. And then we loaded up a... Win... Debug. Spolsky: Windbg! Yeah Atwood: ...and then some .Net managed extensions you can, sort of load. You need like a chi-chi to figure out what the commands are. And then you load the dump, and you load the manage tools. And then you can sort of just investigate all of the threads. You can take the "Show me all the manage threads." And then say "Show me what's the call stack was for that thread." And what we saw was like tons and tons of threads that were all going "Hey, I would like to log something..." And it was like "Hmmmm... [laughs]... Interesting!" Right, you have like 80 threads that all try to write something to the log. So... Right then we kind of knew where the problem was. And then somebody on Twitter actually volunteered to help us diagnose the dump. So I put it up on our server, and he a.... he nailed... he had a great description of it, like line by line, blow by blow of exactly what was happening. I mean, I'm... I'm competent enough to sort of figure out roughly what was going on, but he really knew this stuff and really helped us out, and I do appreciate that. [3:33] Spolsky: That's really awesome. Atwood: So yeah. We.. we ripped out all the logging. I was never a huge fan of logging. I mean I guess there's a couple of philosophies on this. Like which is your philosophy with logging, like as you sit down to write a function, like would you add logging to it? I mean what's your ..what's your philosophy ? I'm curious. [[??]] Spolsky: I'd never .. no ... I never do... I nev... But you know I don't... I don't think I've ever worked on code that is sort of [operational] in the same way. Atwood: ah hm. Spolsky: Eh.. because we definitly eh.. put a lot more ... oh [you know] I did, at Yuno we used to have all kinds of logging. The trouble is that my philosophy has always been that you .. you.. you have a tendency to wanna log everything. But then you just get logs that are, you know, a hundred megabyte per user and you get thirty of them a minute and it can't possibly be analyzed or stored in any reasonable way. So the next thing you have to do is to start culling your logs or just have different levels of debugging, where it's like in high debug mode everything is logged and in low debug mode nothing is logged. And... it's kind of hard to figure out what you really want in a log. You you know you know .. a lot of logs, like I think of the logging that we did in Yuno, where people would call with a complaint and you try to figure out where this programm is crashing. And obviously a log of the crash, that's easy. Ehm, but then there's some line above the crash which hopefully gives you a lot of information about where it happend. And there's some line you don't see that should have been after that, after the crash, but it never got there 'cause it crashed sometime before there. And essentially what you're doing as you're adding logging, is you're doing binary search, right, where you're [sticking in] like "well gosh, I got to here and then got to there. But there's an awful lot of code between point A and point B. So let's make an A you know half-way from A to B, log point of some sort". Then you put that in and then you eliminate 50 percent of the possible places to look for your crash. Um, but I've never really been able to... Atwood: I mean that, ironically, to troubleshoot this hang, which turned out to be because of logging, we were adding more logging. Spolsky: [laughs] Atwood: The joke just writes itself! The joke just writes itself, right... Spolsky: It does... How many... How many third-party tools do you have... uhh... How many third-party tools are a part of the StackOverflow code base? Atwood: Well, okay, so... [chuckles] Uh, Dare [pronounces it as the English word "dare"] Obasanjo [pronounces it "oh-bih-san-ho"]... I don't know if I'm pronouncing it correctly. Spolsky: Okay, "Dare" [pronounces it "daray"]... Obasanjo [pronounces it "oh-bih-san-ja"]... It's "Dare." Atwood: Is it "Dare"? Spolsky: Yep. Atwood: Really... Okay, I didn't know that. Well, I've learned something. But he had a whole blog entry about how, you know, I had chosen to write my own sanitizer, and that was a very deliberate choice for me... Spolsky: Mm hmm. Atwood: ...for a number of reasons that I won't get into. But he was very critical of this, because, of course there were bugs in the sanitizer... Spolsky: Mm hmm. Atwood: ...which there were going to be, and to me, it's about, like, it's about your velocity; it's not about where you are; it's about where you're going, and we're gonna fix that stuff, right, and I'm making the sanitizer public as well, so other people can have a sanitizer that's not ten thousand lines of code, and ridiculous, and uh, so there's a philosophy there of building something that's reusable for everyone. Um, but I thought it was ironic, because he was talking about how developers should just pick a third-party library and go with it, and I think obvio... it's a balancing act, because we picked this logging library, right, which kind of caused a problem for us, right, I mean partially it was the way we were using it, but the way it was logging the files was a design issue in terms of the way it logged for networks. Spolsky: Right. Atwood: So I... I think it's a trade-off. I don't think it's always as clear-cut as "you should always pick a library" or "you should never pick a library," right? I think there's always some in-between there. So, for us, I'm definitely a minimalist—I don't like third-party libraries; I feel like we have a giant third-party library called "Windows," called ".NET"... huh... ASP.NET MVC is technically a third-party library. Um, but these are, you know, major vendor stacks. And I do feel like—as much as we talk about open source and stuff—there's a certain level of quality you associate with these major first-party stacks, right, whether it's from Apple or Microsoft or Sun or whoever. That may or may not be true, but hopefully usually is true: that these things are really heavily tested. Spolsky: There is definitely, yeah, there is definitely... I mean, there's something I've learned over the years, and, you know, I started out with working on the Excel team, um... The developers on that team had a motto, which was "Find the dependencies and eliminate them." You know, they had their own compiler; they would not use untested libraries from other groups at Microsoft even... Atwood: I love that they had their own compiler. That is so hardcore. I can't even, like, I could not even hang out with those guys... right... that hardcore. Spolsky: Hey, well, we have our own compiler, man. Atwood: Yeah... Spolsky: Let me tell you why they had their own compiler: They had their own compiler because Excel was getting huge, and just compiled 8086 was just too large to fit on floppy disks and to fit in memory. You know, we were really trying to cram things in there. And so they developed a pcode compiler, which basically... you know, it's like bytecode. They called it pcode. This is a very old technique, and it compiled Excel into an imaginary machine, a virtual machine, which was a lot more expressive that an 8086, and had all kinds of additional features, and so the compiled code is about one-third the size, and in a lot of situations this made the performance a lot faster. So, for example, in those days when almost everybody was running programs off of floppy disks, the chance is... Or no, not floppy, but the 3.5-inch, not-so-floppy disks. But the read time on those things is really really slow, so if you could launch your app—if your app was smaller at the time that you read it from disk—it didn't matter if it ran a little bit slower. The whole... the overall experience would be a lot faster. So if you could fit in memory without swapping, then obviously the whole thing would run faster, so it was worth doing this pcode thing for a long time, and about the time of Excel 5.0, the bit flipped on that and it suddenly became... suddenly everybody had hard drives, and nobody really cared about the size of the executable, and it was okay to have about a, I think, a four-megabyte executable instead of a one-megabyte executable, and so they got rid of that pcode back-end. But even then I think they had their own compiler for a while because in order to right really really efficient code, they wanted to be able to control... Now this is a long story. [9:39] Spolsky: but a pointer on an 80386... for a while the 80386 was the target. On an 80386, a pointer consisted of (or even on the 8086 in general) a pointer consisted of two parts, the segment and the offset. So it's like "where do you want to start your pointer?" and then "what's your offset inside there?". And you couldn't just indirectly say "here's my pointer, just do something with this" Atwood: edit me! Spolsky: edit me! [To be transcribed] [20:26] Atwood: But people who do something like that, they don't want attention or anything, their not feed on it all, their not interesting in things like attention at all. The work is it's own reward, it doesn't matter if anyone is looking. Have you seen that thing on the ... I meant to blog about this but the whole concept of just, not looking at things, to basically discourage them. Or that, conversely, looking at things encourages them, like the whole Paris Hilton thing and just talking about these things over and over incessantly actually reinforces the whole trend. There was a series of children's books I don't know if you've heard them, their called: "The Great Brain". It's set in Utah - Spolsky: Yeah Atwood: - at the turn of the century. I got these books as a kid and I was totally obsessed with them, because the Great Brain is all about a family. I don't remember the family's name, but there is one central character J.D. who is the Great Brain. Essentially he's always thinking up ways to, essentially social engineering before we had that word in computer circles. Basically getting people to do what you want them to do completely of their own volition. The Great Brain is basically this genius of a kid, who is using all these social engineering exploits to get away with all this crazy stuff. In that family, if they found out the Great Brain was doing this stuff then of course he would get punished, but the ultimate penalty was what they called the silent treatment. The silent treatment meant that nobody would talk to you or acknowledge you for a certain period of time. They would give you food and stuff, but they wouldn't talk at you. It was just stunning in the book, you don't really think about this stuff as a kid (I was like 10 or so, how old I was), how desperate it is as a person, as a social being, when nobody will acknowledge you. How profoundly affecting that is, right? Even the Great Brain, as a smart kid, hated the silent treatment and would do anything he could to avoid getting the silent treatment because it was just such a brutal penalty. I remember Jason Cotke (sp?) was talking about an episode of The Simpsons where these animated statues came to life, and the way they got rid of them was they started chanting "Just Don't Look!" Spolsky: (laughs) Atwood: This is led by Lisa Simpson, who says "Just don't look at them and they'll go away!" Spolsky: (laughs) Atwood: Its amazing how powerful that philosophy is. If there are things happening you don't like just don't talk about it or don't give it any attention, and look at the things that you actually care about and actually want (to have happen). Spolsky: Thats right. That was like the Clinton administration's policy on Rwanda. Atwood: (uncertain chuckle) Well thats the issue of social injustices, which I think is a little bit different. Spolsky: There are definitely people who are attention seeking, but you know what? That the thing about trolls, "don't feed the trolls" or whatever. You know what, trolls are doing a great service to the Internet, they're making it entertaining and interesting. Atwood: Absolutely. [23:21] [1:02:50] Atwood: Two things; we have a wiki, for people who can't listen to this. Where people can contribute transcriptions of our incredibly boring podcasts, and we thank you very much for that. Although I do have one request for the transcriptionists, and the ironic thing is, you're going to transcribe this, which I think is hilarious. When you transcribe, don't write down every time I say 'uh' or pause or 'yeah'. Make me sound awesome, that's my one request for the transcriptionists. Spolsky: It doesn't have to be word for word. It doesn't necessarily read as well when it's word for word you can leave [out] 'uhms' and 'uhs'. Atwood: In fact, leave out whole. If you think it reads better a certain way, just make me say whatever makes the transcriptionists sound the most awesome. Spolsky: And it's wiki, go ahead and edit it. Atwood: People edit anyway, you're right, it's hilarious. I've been reading the [revisions], it's very funny. Atwood: The other thing is, if you do contribute to the wiki. Since our beta has been pushed back a week, this will get you in the same day, to the StackOverflow beta. If you want to be in, just email me after you've done a little bit of transcription, one minute or whatever you're comfortable with. If you want to get your question answered on the air, send a less than 90 second recording to podcast@stackoverflow.com, we will put it in the queue and hopefully answer it on the next podcast. Spolsky: Alright, that's it. Thank you very much, see you next week! Atwood: See you next week. [1:04:04] [Outro]	Ads, intro [1:00] Spolsky: Today is the day that we did not launch, although we planned to. But then... We'll wait for another week. Atwood: Yeah, well, the good news on that is that we did actually figure out what that problem was. Spolsky: Oh, oh, I want to hear, I want to hear, I want to hear. Atwood: Eh. So, it was a third party library. Indirectly, I mean. It's the third party library, and our particular use of it. It was Log4Net. Spolsky: Oh! Atwood: We were logging in such a way that the log.... during the log call was triggering another log call. Which is normally okay, but with the load that we have, eventually they would happen so close together that there's also a lock. So, there's two locks going on there. There's a lock of like disposing of the database stuff that's going on. Then there's lock of like actually writing to a file... Spolsky: Hm! Atwood: And... Huh... They happen in the opposite order, so it's like a classic deadlock, right. So, you release the lock on the database, then you release the lock on the file. And then the other call was doing in the other order. And they were happening so fast that... it was deadlocking eventually. And it was one of those things that would happen.. like... it was very [intermittent [??]], right. Spolsky: Right. Atwood: So we had to dust-off Win Debug. Spolsky: How on Earth do you find things like that? Atwood: Well. You bust that Win Debug. One nice feature in Windows 2008, and I think this is in Vista as well. In Task Manager, you can right-click a task and take a dump of it. Spolsky: Yeah! Atwood: Like right there. Spolsky: Aha. Atwood: So we took a dump of the W3Service process, and... Spolsky: [Then [??]] take a dump. Atwood: Yeah, I know, any time you do this it's like.. It's like the territory for jokes. It's just... Spolsky: [giggles] Atwood: [laughs]. And then we loaded up a... Win... Debug. Spolsky: Windbg! Yeah Atwood: ...and then some .Net managed extensions you can, sort of load. You need like a chi-chi to figure out what the commands are. And then you load the dump, and you load the manage tools. And then you can sort of just investigate all of the threads. You can take the "Show me all the manage threads." And then say "Show me what's the call stack was for that thread." And what we saw was like tons and tons of threads that were all going "Hey, I would like to log something..." And it was like "Hmmmm... [laughs]... Interesting!" Right, you have like 80 threads that all try to write something to the log. So... Right then we kind of knew where the problem was. And then somebody on Twitter actually volunteered to help us diagnose the dump. So I put it up on our server, and he a.... he nailed... he had a great description of it, like line by line, blow by blow of exactly what was happening. I mean, I'm... I'm competent enough to sort of figure out roughly what was going on, but he really knew this stuff and really helped us out, and I do appreciate that. [3:33] Spolsky: That's really awesome. Atwood: So yeah. We.. we ripped out all the logging. I was never a huge fan of logging. I mean I guess there's a couple of philosophies on this. Like which is your philosophy with logging, like as you sit down to write a function, like would you add logging to it? I mean what's your ..what's your philosophy ? I'm curious. [[??]] Spolsky: I'd never .. no ... I never do... I nev... But you know I don't... I don't think I've ever worked on code that is sort of [operational] in the same way. Atwood: ah hm. Spolsky: Eh.. because we definitely eh.. put a lot more ... oh [you know] I did, at Yuno we used to have all kinds of logging. The trouble is that my philosophy has always been that you .. you.. you have a tendency to wanna log everything. But then you just get logs that are, you know, a hundred megabyte per user and you get thirty of them a minute and it can't possibly be analyzed or stored in any reasonable way. So the next thing you have to do is to start culling your logs or just have different levels of debugging, where it's like in high debug mode everything is logged and in low debug mode nothing is logged. And... it's kind of hard to figure out what you really want in a log. You you know you know .. a lot of logs, like I think of the logging that we did in Yuno, where people would call with a complaint and you try to figure out where this program is crashing. And obviously a log of the crash, that's easy. Ehm, but then there's some line above the crash which hopefully gives you a lot of information about where it happened. And there's some line you don't see that should have been after that, after the crash, but it never got there 'cause it crashed sometime before there. And essentially what you're doing as you're adding logging, is you're doing binary search, right, where you're [sticking in] like "well gosh, I got to here and then got to there. But there's an awful lot of code between point A and point B. So let's make an A you know half-way from A to B, log point of some sort". Then you put that in and then you eliminate 50 percent of the possible places to look for your crash. Um, but I've never really been able to... Atwood: I mean that, ironically, to troubleshoot this hang, which turned out to be because of logging, we were adding more logging. Spolsky: [laughs] Atwood: The joke just writes itself! The joke just writes itself, right... Spolsky: It does... How many... How many third-party tools do you have... uhh... How many third-party tools are a part of the StackOverflow code base? Atwood: Well, okay, so... [chuckles] Uh, Dare [pronounces it as the English word "dare"] Obasanjo [pronounces it "oh-bih-san-ho"]... I don't know if I'm pronouncing it correctly. Spolsky: Okay, "Dare" [pronounces it "daray"]... Obasanjo [pronounces it "oh-bih-san-ja"]... It's "Dare." Atwood: Is it "Dare"? Spolsky: Yep. Atwood: Really... Okay, I didn't know that. Well, I've learned something. But he had a whole blog entry about how, you know, I had chosen to write my own sanitizer, and that was a very deliberate choice for me... Spolsky: Mm hmm. Atwood: ...for a number of reasons that I won't get into. But he was very critical of this, because, of course there were bugs in the sanitizer... Spolsky: Mm hmm. Atwood: ...which there were going to be, and to me, it's about, like, it's about your velocity; it's not about where you are; it's about where you're going, and we're gonna fix that stuff, right, and I'm making the sanitizer public as well, so other people can have a sanitizer that's not ten thousand lines of code, and ridiculous, and uh, so there's a philosophy there of building something that's reusable for everyone. Um, but I thought it was ironic, because he was talking about how developers should just pick a third-party library and go with it, and I think obvio... it's a balancing act, because we picked this logging library, right, which kind of caused a problem for us, right, I mean partially it was the way we were using it, but the way it was logging the files was a design issue in terms of the way it logged for networks. Spolsky: Right. Atwood: So I... I think it's a trade-off. I don't think it's always as clear-cut as "you should always pick a library" or "you should never pick a library," right? I think there's always some in-between there. So, for us, I'm definitely a minimalist—I don't like third-party libraries; I feel like we have a giant third-party library called "Windows," called ".NET"... huh... ASP.NET MVC is technically a third-party library. Um, but these are, you know, major vendor stacks. And I do feel like—as much as we talk about open source and stuff—there's a certain level of quality you associate with these major first-party stacks, right, whether it's from Apple or Microsoft or Sun or whoever. That may or may not be true, but hopefully usually is true: that these things are really heavily tested. Spolsky: There is definitely, yeah, there is definitely... I mean, there's something I've learned over the years, and, you know, I started out with working on the Excel team, um... The developers on that team had a motto, which was "Find the dependencies and eliminate them." You know, they had their own compiler; they would not use untested libraries from other groups at Microsoft even... Atwood: I love that they had their own compiler. That is so hardcore. I can't even, like, I could not even hang out with those guys... right... that hardcore. Spolsky: Hey, well, we have our own compiler, man. Atwood: Yeah... Spolsky: Let me tell you why they had their own compiler: They had their own compiler because Excel was getting huge, and just compiled 8086 was just too large to fit on floppy disks and to fit in memory. You know, we were really trying to cram things in there. And so they developed a pcode compiler, which basically... you know, it's like bytecode. They called it pcode. This is a very old technique, and it compiled Excel into an imaginary machine, a virtual machine, which was a lot more expressive that an 8086, and had all kinds of additional features, and so the compiled code is about one-third the size, and in a lot of situations this made the performance a lot faster. So, for example, in those days when almost everybody was running programs off of floppy disks, the chance is... Or no, not floppy, but the 3.5-inch, not-so-floppy disks. But the read time on those things is really really slow, so if you could launch your app—if your app was smaller at the time that you read it from disk—it didn't matter if it ran a little bit slower. The whole... the overall experience would be a lot faster. So if you could fit in memory without swapping, then obviously the whole thing would run faster, so it was worth doing this pcode thing for a long time, and about the time of Excel 5.0, the bit flipped on that and it suddenly became... suddenly everybody had hard drives, and nobody really cared about the size of the executable, and it was okay to have about a, I think, a four-megabyte executable instead of a one-megabyte executable, and so they got rid of that pcode back-end. But even then I think they had their own compiler for a while because in order to right really really efficient code, they wanted to be able to control... Now this is a long story. [9:39] Spolsky: but a pointer on an 80386... for a while the 80386 was the target. On an 80386, a pointer consisted of (or even on the 8086 in general) a pointer consisted of two parts, the segment and the offset. So it's like "where do you want to start your pointer?" and then "what's your offset inside there?". And you couldn't just indirectly say "here's my pointer, just do something with this" Atwood: edit me! Spolsky: edit me! [To be transcribed] [20:26] Atwood: But people who do something like that, they don't want attention or anything, their not feed on it all, their not interesting in things like attention at all. The work is it's own reward, it doesn't matter if anyone is looking. Have you seen that thing on the ... I meant to blog about this but the whole concept of just, not looking at things, to basically discourage them. Or that, conversely, looking at things encourages them, like the whole Paris Hilton thing and just talking about these things over and over incessantly actually reinforces the whole trend. There was a series of children's books I don't know if you've heard them, their called: "The Great Brain". It's set in Utah - Spolsky: Yeah Atwood: - at the turn of the century. I got these books as a kid and I was totally obsessed with them, because the Great Brain is all about a family. I don't remember the family's name, but there is one central character J.D. who is the Great Brain. Essentially he's always thinking up ways to, essentially social engineering before we had that word in computer circles. Basically getting people to do what you want them to do completely of their own volition. The Great Brain is basically this genius of a kid, who is using all these social engineering exploits to get away with all this crazy stuff. In that family, if they found out the Great Brain was doing this stuff then of course he would get punished, but the ultimate penalty was what they called the silent treatment. The silent treatment meant that nobody would talk to you or acknowledge you for a certain period of time. They would give you food and stuff, but they wouldn't talk at you. It was just stunning in the book, you don't really think about this stuff as a kid (I was like 10 or so, how old I was), how desperate it is as a person, as a social being, when nobody will acknowledge you. How profoundly affecting that is, right? Even the Great Brain, as a smart kid, hated the silent treatment and would do anything he could to avoid getting the silent treatment because it was just such a brutal penalty. I remember Jason Cotke (sp?) was talking about an episode of The Simpsons where these animated statues came to life, and the way they got rid of them was they started chanting "Just Don't Look!" Spolsky: (laughs) Atwood: This is led by Lisa Simpson, who says "Just don't look at them and they'll go away!" Spolsky: (laughs) Atwood: Its amazing how powerful that philosophy is. If there are things happening you don't like just don't talk about it or don't give it any attention, and look at the things that you actually care about and actually want (to have happen). Spolsky: Thats right. That was like the Clinton administration's policy on Rwanda. Atwood: (uncertain chuckle) Well thats the issue of social injustices, which I think is a little bit different. Spolsky: There are definitely people who are attention seeking, but you know what? That the thing about trolls, "don't feed the trolls" or whatever. You know what, trolls are doing a great service to the Internet, they're making it entertaining and interesting. Atwood: Absolutely. [23:21] [57:56] Spolsky: [We will actually have] I'm in the process of making another movie. Did we talking about this on the podcast yet? Atwood: Oh really? You sent me a copy of Aardvark'd. I have seen that. Spolsky: That was our first movie. That was like a slightly serious reality TV show about the interns building Copilot. It was probably better than what on MTV. But the truth is the audience, the Joel on Software audience, really wanted something more technical and a lot of people wrote to me and said "You know, I really wanted to learn how software is developed at Fog Creek", and this movie was just a little bit too light on that content. This is going to be much more hardcore. SO it's really going to be about how software is developed at Fog Creek. Atwood: Ooh wow... Has this started? Spolsky: It's about a one year project. We've got the same film maker as we had last time, except I'm basically interviewing everybody at Fog Creek. You know, three or four times, to get footage for this thing. And then we'll splice it all together, and the current plan is to have several different formats of this movie. There'll be a lot of short form little piece, like four minutes, we can put up on YouTube. If you want to hear about source code control, or you want to learn about hiring programmers or interviewing programmers or the phone interview. They'll be these little pieces that you can just download freely on the internet or watch. And then there's a [sort of a] more substantial ninety minute version, which is probably too much to distribute over the internet. But it'll probably distribute in DVD form. Maybe by that time, there'll be a good way to distribute a [you know] ninety minute DVD length thing over the internet. At kind of the high end, and what will hopefully pay for this whole operation, will be a corporate training video. Maybe a five to six hour thing that a team might buy inside a corporation to learn how to do software development a little bit better, and it'll just go into much more depth. There'll be like a whole hour of training on how to conduct interviews for programmers and a whole hour on how to setup your tools and that kind of stuff. Atwood: Wow. So you're like Howard Stern. You're like the kind of the media now. You have like your movie. You've got your podcasts with me. You've got, you know, your blog, your not-a-blog. Spolsky: But I mean, you know, that was the original idea behind this podcast remember how I said, like, "There's some people that you can reach in certain channels, and they need to hear you. I don't listen to Eminem's music, blog I saw the movie 8-mile, or miles... whatever it was. Different people, you will reach in different channels. Part of the goal is to do that, and the other part of the goal is: I have an inherent inability to ever give up. And instead of just saying, "You know, we tried to make a movie, and it wasn't as good as it should have been, and so maybe we're not so good at making movies.", we just said "You know, let's learn from the experience. Let's try and make a two point oh, that's just much better." Atwood: I didn't really hear a lot of negative stuff about it. Spolsky: You know who liked it? Programmer's girlfriend, wives and families. Because there's nothing about it that's inaccessible to somebody who knows nothing about programming. And it really does convey to the normal person a lot about what a programmer's life is like. Just the sort of feel or being a programmer. The fact that programmers get into these conversations about things. You know that was probably, I thought, one of the highlights of the movie - when we told the interns that their job was to figure out if they could jump across to the ledge of the neighboring building if there was a fire. Could they jump out the window and make it on to the ledge of the neighboring building? And programmers love to have conversations like that, and they immediately leapt up to the white board and started drawing equations from physics and stuff like that, and they tried to work it out. And that's really what our conversations are like, what our lives are like. And to geeks, nobody knows--, if you're a geek you probably don't think anything is strange about this. But to a normal human being... this is very bizarre. That we talk that way, and that we think about these kind of things, and that we then need to solve the problem mathematically, to decide if you can make it to the neighboring ledge before you died. Atwood: Right... Well that's cool. I'm looking forward to that coming out. Looks like it'll be a while. I definitely recommend people who haven't seen it, check out Aardvark'd. I enjoyed it. I didn't think it lacked anything. Spolsky: The way to find that is ProjectAardvark.com, and there's link to the movie. ProjectAardvark.com is a blog that the interns kept that summer. It was a couple of years ago. [1:02:14] [1:02:50] Atwood: Two things; we have a wiki, for people who can't listen to this. Where people can contribute transcriptions of our incredibly boring podcasts, and we thank you very much for that. Although I do have one request for the transcriptionists, and the ironic thing is, you're going to transcribe this, which I think is hilarious. When you transcribe, don't write down every time I say 'uh' or pause or 'yeah'. Make me sound awesome, that's my one request for the transcriptionists. Spolsky: It doesn't have to be word for word. It doesn't necessarily read as well when it's word for word you can leave [out] 'uhms' and 'uhs'. Atwood: In fact, leave out whole. If you think it reads better a certain way, just make me say whatever makes the transcriptionists sound the most awesome. Spolsky: And it's wiki, go ahead and edit it. Atwood: People edit anyway, you're right, it's hilarious. I've been reading the [revisions], it's very funny. Atwood: The other thing is, if you do contribute to the wiki. Since our beta has been pushed back a week, this will get you in the same day, to the StackOverflow beta. If you want to be in, just email me after you've done a little bit of transcription, one minute or whatever you're comfortable with. If you want to get your question answered on the air, send a less than 90 second recording to podcast@stackoverflow.com, we will put it in the queue and hopefully answer it on the next podcast. Spolsky: Alright, that's it. Thank you very much, see you next week! Atwood: See you next week. [1:04:04] [Outro]

Ads, intro

[1:00]

Spolsky: Today is the day that we did not launch, although we planned to. But then... We'll wait for another week.

Atwood: Yeah, well, the good news on that is that we did actually figure out what that problem was.

Spolsky: Oh, oh, I want to hear, I want to hear, I want to hear.

Atwood: Eh. So, it was a third party library. Indirectly, I mean. It's the third party library, and our particular use of it. It was Log4Net.

Spolsky: Oh!

Atwood: We were logging in such a way that the log.... during the log call was triggering another log call. Which is normally okay, but with the load that we have, eventually they would happen so close together that there's also a lock. So, there's two locks going on there. There's a lock of like disposing of the database stuff that's going on. Then there's lock of like actually writing to a file...

Spolsky: Hm!

Atwood: And... Huh... They happen in the opposite order, so it's like a classic deadlock, right. So, you release the lock on the database, then you release the lock on the file. And then the other call was doing in the other order. And they were happenning so fast that... it was deadlocking eventually. And it was one of those things that would happen.. like... it was very [intermittent [??]], right.

Spolsky: Right.

Atwood: So we had to dust-off Win Debug.

Spolsky: How on Earth do you find things like that?

Atwood: Well. You bust that Win Debug. One nice feature in Windows 2008, and I think this is in Vista as well. In Task Manager, you can right-click a task and take a dump of it.

Spolsky: Yeah!

Atwood: Like right there.

Spolsky: Aha.

Atwood: So we took a dump of the W3Service process, and...

Spolsky: [Then [??]] take a dump.

Atwood: Yeah, I know, any time you do this it's like.. It's like the territory for jokes. It's just...

Spolsky: [giggles]

Atwood: [laughs]. And then we loaded up a... Win... Debug.

Spolsky: Windbg! Yeah

Atwood: ...and then some .Net managed extensions you can, sort of load. You need like a chi-chi to figure out what the commands are. And then you load the dump, and you load the manage tools. And then you can sort of just investigate all of the threads. You can take the "Show me all the manage threads." And then say "Show me what's the call stack was for that thread." And what we saw was like tons and tons of threads that were all going "Hey, I would like to log something..." And it was like "Hmmmm... [laughs]... Interesting!" Right, you have like 80 threads that all try to write something to the log. So... Right then we kind of knew where the problem was.

And then somebody on Twitter actually volunteered to help us diagnose the dump. So I put it up on our server, and he a.... he nailed... he had a great description of it, like line by line, blow by blow of exactly what was happening. I mean, I'm... I'm competent enough to sort of figure out roughly what was going on, but he really knew this stuff and really helped us out, and I do appreciate that.

[3:33]

Spolsky: That's really awesome.

Atwood: So yeah. We.. we ripped out all the logging. I was never a huge fan of logging. I mean I guess there's a couple of philosophies on this. Like which is your philosophy with logging, like as you sit down to write a function, like would you add logging to it? I mean what's your ..what's your philosophy ? I'm curious. [[??]]

Spolsky: I'd never .. no ... I never do... I nev... But you know I don't... I don't think I've ever worked on code that is sort of [operational] in the same way.

Atwood: ah hm.

Spolsky: Eh.. because we definitly eh.. put a lot more ... oh [you know] I did, at Yuno we used to have all kinds of logging.
The trouble is that my philosophy has always been that you .. you.. you have a tendency to wanna log everything. But then you just get logs that are, you know, a hundred megabyte per user and you get thirty of them a minute and it can't possibly be analyzed or stored in any reasonable way. So the next thing you have to do is to start culling your logs or just have different levels of debugging, where it's like in high debug mode everything is logged and in low debug mode nothing is logged. And... it's kind of hard to figure out what you really want in a log. You you know you know .. a lot of logs, like I think of the logging that we did in Yuno, where people would call with a complaint and you try to figure out where this programm is crashing. And obviously a log of the crash, that's easy. Ehm, but then there's some line above the crash which hopefully gives you a lot of information about where it happend. And there's some line you don't see that should have been after that, after the crash, but it never got there 'cause it crashed sometime before there. And essentially what you're doing as you're adding logging, is you're doing binary search, right, where you're [sticking in] like "well gosh, I got to here and then got to there. But there's an awful lot of code between point A and point B. So let's make an A you know half-way from A to B, log point of some sort". Then you put that in and then you eliminate 50 percent of the possible places to look for your crash.
Um, but I've never really been able to...

Atwood: I mean that, ironically, to troubleshoot this hang, which turned out to be because of logging, we were adding more logging.

Spolsky: [laughs]

Atwood: The joke just writes itself! The joke just writes itself, right...

Spolsky: It does... How many... How many third-party tools do you have... uhh... How many third-party tools are a part of the StackOverflow code base?

Atwood: Well, okay, so... [chuckles] Uh, Dare [pronounces it as the English word "dare"] Obasanjo [pronounces it "oh-bih-san-ho"]... I don't know if I'm pronouncing it correctly.

Spolsky: Okay, "Dare" [pronounces it "daray"]... Obasanjo [pronounces it "oh-bih-san-ja"]... It's "Dare."

Atwood: Is it "Dare"?

Spolsky: Yep.

Atwood: Really... Okay, I didn't know that. Well, I've learned something. But he had a whole blog entry about how, you know, I had chosen to write my own sanitizer, and that was a very deliberate choice for me...

Spolsky: Mm hmm.

Atwood: ...for a number of reasons that I won't get into. But he was very critical of this, because, of course there were bugs in the sanitizer...

Spolsky: Mm hmm.

Atwood: ...which there were going to be, and to me, it's about, like, it's about your velocity; it's not about where you are; it's about where you're going, and we're gonna fix that stuff, right, and I'm making the sanitizer public as well, so other people can have a sanitizer that's not ten thousand lines of code, and ridiculous, and uh, so there's a philosophy there of building something that's reusable for everyone. Um, but I thought it was ironic, because he was talking about how developers should just pick a third-party library and go with it, and I think obvio... it's a balancing act, because we picked this logging library, right, which kind of caused a problem for us, right, I mean partially it was the way we were using it, but the way it was logging the files was a design issue in terms of the way it logged for networks.

Spolsky: Right.

Atwood: So I... I think it's a trade-off. I don't think it's always as clear-cut as "you should always pick a library" or "you should never pick a library," right? I think there's always some in-between there. So, for us, I'm definitely a minimalist—I don't like third-party libraries; I feel like we have a giant third-party library called "Windows," called ".NET"... huh... ASP.NET MVC is technically a third-party library. Um, but these are, you know, major vendor stacks. And I do feel like—as much as we talk about open source and stuff—there's a certain level of quality you associate with these major first-party stacks, right, whether it's from Apple or Microsoft or Sun or whoever. That may or may not be true, but hopefully usually is true: that these things are really heavily tested.

Spolsky: There is definitely, yeah, there is definitely... I mean, there's something I've learned over the years, and, you know, I started out with working on the Excel team, um... The developers on that team had a motto, which was "Find the dependencies and eliminate them." You know, they had their own compiler; they would not use untested libraries from other groups at Microsoft even...

Atwood: I love that they had their own compiler. That is so hardcore. I can't even, like, I could not even hang out with those guys... right... that hardcore.

Spolsky: Hey, well, we have our own compiler, man.

Atwood: Yeah...

Spolsky: Let me tell you why they had their own compiler: They had their own compiler because Excel was getting huge, and just compiled 8086 was just too large to fit on floppy disks and to fit in memory. You know, we were really trying to cram things in there. And so they developed a pcode compiler, which basically... you know, it's like bytecode. They called it pcode. This is a very old technique, and it compiled Excel into an imaginary machine, a virtual machine, which was a lot more expressive that an 8086, and had all kinds of additional features, and so the compiled code is about one-third the size, and in a lot of situations this made the performance a lot faster. So, for example, in those days when almost everybody was running programs off of floppy disks, the chance is... Or no, not floppy, but the 3.5-inch, not-so-floppy disks. But the read time on those things is really really slow, so if you could launch your app—if your app was smaller at the time that you read it from disk—it didn't matter if it ran a little bit slower. The whole... the overall experience would be a lot faster. So if you could fit in memory without swapping, then obviously the whole thing would run faster, so it was worth doing this pcode thing for a long time, and about the time of Excel 5.0, the bit flipped on that and it suddenly became... suddenly everybody had hard drives, and nobody really cared about the size of the executable, and it was okay to have about a, I think, a four-megabyte executable instead of a one-megabyte executable, and so they got rid of that pcode back-end. But even then I think they had their own compiler for a while because in order to right really really efficient code, they wanted to be able to control... Now this is a long story.

[9:39]

Spolsky: but a pointer on an 80386... for a while the 80386 was the target. On an 80386, a pointer consisted of (or even on the 8086 in general) a pointer consisted of two parts, the segment and the offset. So it's like "where do you want to start your pointer?" and then "what's your offset inside there?". And you couldn't just indirectly say "here's my pointer, just do something with this"

Atwood: edit me!

Spolsky: edit me!

[To be transcribed]

[20:26]
Atwood: But people who do something like that, they don't want attention or anything, their not feed on it all, their not interesting in things like attention at all. The work is it's own reward, it doesn't matter if anyone is looking. Have you seen that thing on the ... I meant to blog about this but the whole concept of just, not looking at things, to basically discourage them. Or that, conversely, looking at things encourages them, like the whole Paris Hilton thing and just talking about these things over and over incessantly actually reinforces the whole trend. There was a series of children's books I don't know if you've heard them, their called: "The Great Brain". It's set in Utah -

Spolsky: Yeah

Atwood: - at the turn of the century. I got these books as a kid and I was totally obsessed with them, because the Great Brain is all about a family. I don't remember the family's name, but there is one central character J.D. who is the Great Brain. Essentially he's always thinking up ways to, essentially social engineering before we had that word in computer circles. Basically getting people to do what you want them to do completely of their own volition. The Great Brain is basically this genius of a kid, who is using all these social engineering exploits to get away with all this crazy stuff. In that family, if they found out the Great Brain was doing this stuff then of course he would get punished, but the ultimate penalty was what they called the silent treatment. The silent treatment meant that nobody would talk to you or acknowledge you for a certain period of time. They would give you food and stuff, but they wouldn't talk at you. It was just stunning in the book, you don't really think about this stuff as a kid (I was like 10 or so, how old I was), how desperate it is as a person, as a social being, when nobody will acknowledge you. How profoundly affecting that is, right? Even the Great Brain, as a smart kid, hated the silent treatment and would do anything he could to avoid getting the silent treatment because it was just such a brutal penalty. I remember Jason Cotke (sp?) was talking about an episode of The Simpsons where these animated statues came to life, and the way they got rid of them was they started chanting "Just Don't Look!"

Spolsky: (laughs)

Atwood: This is led by Lisa Simpson, who says "Just don't look at them and they'll go away!"

Spolsky: (laughs)

Atwood: Its amazing how powerful that philosophy is. If there are things happening you don't like just don't talk about it or don't give it any attention, and look at the things that you actually care about and actually want (to have happen).

Spolsky: Thats right. That was like the Clinton administration's policy on Rwanda.

Atwood: (uncertain chuckle) Well thats the issue of social injustices, which I think is a little bit different.

Spolsky: There are definitely people who are attention seeking, but you know what? That the thing about trolls, "don't feed the trolls" or whatever. You know what, trolls are doing a great service to the Internet, they're making it entertaining and interesting.

Atwood: Absolutely.

[23:21]

[1:02:50]

Atwood: Two things; we have a wiki, for people who can't listen to this. Where people can contribute transcriptions of our incredibly boring podcasts, and we thank you very much for that. Although I do have one request for the transcriptionists, and the ironic thing is, you're going to transcribe this, which I think is hilarious. When you transcribe, don't write down every time I say 'uh' or pause or 'yeah'. Make me sound awesome, that's my one request for the transcriptionists.

Spolsky: It doesn't have to be word for word. It doesn't necessarily read as well when it's word for word you can leave [out] 'uhms' and 'uhs'.

Atwood: In fact, leave out whole. If you think it reads better a certain way, just make me say whatever makes the transcriptionists sound the most awesome.

Spolsky: And it's wiki, go ahead and edit it.

Atwood: People edit anyway, you're right, it's hilarious. I've been reading the [revisions], it's very funny.

Atwood: The other thing is, if you do contribute to the wiki. Since our beta has been pushed back a week, this will get you in the same day, to the StackOverflow beta. If you want to be in, just email me after you've done a little bit of transcription, one minute or whatever you're comfortable with. If you want to get your question answered on the air, send a less than 90 second recording to podcast@stackoverflow.com, we will put it in the queue and hopefully answer it on the next podcast.

Spolsky: Alright, that's it. Thank you very much, see you next week!

Atwood: See you next week.

[1:04:04]

[Outro]

Ads, intro

[1:00]

Spolsky: Today is the day that we did not launch, although we planned to. But then... We'll wait for another week.

Atwood: Yeah, well, the good news on that is that we did actually figure out what that problem was.

Spolsky: Oh, oh, I want to hear, I want to hear, I want to hear.

Atwood: Eh. So, it was a third party library. Indirectly, I mean. It's the third party library, and our particular use of it. It was Log4Net.

Spolsky: Oh!

Spolsky: Hm!

Atwood: And... Huh... They happen in the opposite order, so it's like a classic deadlock, right. So, you release the lock on the database, then you release the lock on the file. And then the other call was doing in the other order. And they were happening so fast that... it was deadlocking eventually. And it was one of those things that would happen.. like... it was very [intermittent [??]], right.

Spolsky: Right.

Atwood: So we had to dust-off Win Debug.

Spolsky: How on Earth do you find things like that?

Atwood: Well. You bust that Win Debug. One nice feature in Windows 2008, and I think this is in Vista as well. In Task Manager, you can right-click a task and take a dump of it.

Spolsky: Yeah!

Atwood: Like right there.

Spolsky: Aha.

Atwood: So we took a dump of the W3Service process, and...

Spolsky: [Then [??]] take a dump.

Atwood: Yeah, I know, any time you do this it's like.. It's like the territory for jokes. It's just...

Spolsky: [giggles]

Atwood: [laughs]. And then we loaded up a... Win... Debug.

Spolsky: Windbg! Yeah

Spolsky: That's really awesome.

Atwood: So yeah. We.. we ripped out all the logging. I was never a huge fan of logging. I mean I guess there's a couple of philosophies on this. Like which is your philosophy with logging, like as you sit down to write a function, like would you add logging to it? I mean what's your ..what's your philosophy ? I'm curious. [[??]]

Spolsky: I'd never .. no ... I never do... I nev... But you know I don't... I don't think I've ever worked on code that is sort of [operational] in the same way.

Atwood: ah hm.

Spolsky: Eh.. because we definitely eh.. put a lot more ... oh [you know] I did, at Yuno we used to have all kinds of logging.
The trouble is that my philosophy has always been that you .. you.. you have a tendency to wanna log everything. But then you just get logs that are, you know, a hundred megabyte per user and you get thirty of them a minute and it can't possibly be analyzed or stored in any reasonable way. So the next thing you have to do is to start culling your logs or just have different levels of debugging, where it's like in high debug mode everything is logged and in low debug mode nothing is logged. And... it's kind of hard to figure out what you really want in a log. You you know you know .. a lot of logs, like I think of the logging that we did in Yuno, where people would call with a complaint and you try to figure out where this program is crashing. And obviously a log of the crash, that's easy. Ehm, but then there's some line above the crash which hopefully gives you a lot of information about where it happened. And there's some line you don't see that should have been after that, after the crash, but it never got there 'cause it crashed sometime before there. And essentially what you're doing as you're adding logging, is you're doing binary search, right, where you're [sticking in] like "well gosh, I got to here and then got to there. But there's an awful lot of code between point A and point B. So let's make an A you know half-way from A to B, log point of some sort". Then you put that in and then you eliminate 50 percent of the possible places to look for your crash.
Um, but I've never really been able to...

Atwood: I mean that, ironically, to troubleshoot this hang, which turned out to be because of logging, we were adding more logging.

Spolsky: [laughs]

Atwood: The joke just writes itself! The joke just writes itself, right...

Spolsky: It does... How many... How many third-party tools do you have... uhh... How many third-party tools are a part of the StackOverflow code base?

Atwood: Well, okay, so... [chuckles] Uh, Dare [pronounces it as the English word "dare"] Obasanjo [pronounces it "oh-bih-san-ho"]... I don't know if I'm pronouncing it correctly.

Spolsky: Okay, "Dare" [pronounces it "daray"]... Obasanjo [pronounces it "oh-bih-san-ja"]... It's "Dare."

Atwood: Is it "Dare"?

Spolsky: Yep.

Atwood: Really... Okay, I didn't know that. Well, I've learned something. But he had a whole blog entry about how, you know, I had chosen to write my own sanitizer, and that was a very deliberate choice for me...

Spolsky: Mm hmm.

Atwood: ...for a number of reasons that I won't get into. But he was very critical of this, because, of course there were bugs in the sanitizer...

Spolsky: Mm hmm.

Atwood: ...which there were going to be, and to me, it's about, like, it's about your velocity; it's not about where you are; it's about where you're going, and we're gonna fix that stuff, right, and I'm making the sanitizer public as well, so other people can have a sanitizer that's not ten thousand lines of code, and ridiculous, and uh, so there's a philosophy there of building something that's reusable for everyone. Um, but I thought it was ironic, because he was talking about how developers should just pick a third-party library and go with it, and I think obvio... it's a balancing act, because we picked this logging library, right, which kind of caused a problem for us, right, I mean partially it was the way we were using it, but the way it was logging the files was a design issue in terms of the way it logged for networks.

Spolsky: Right.

Atwood: So I... I think it's a trade-off. I don't think it's always as clear-cut as "you should always pick a library" or "you should never pick a library," right? I think there's always some in-between there. So, for us, I'm definitely a minimalist—I don't like third-party libraries; I feel like we have a giant third-party library called "Windows," called ".NET"... huh... ASP.NET MVC is technically a third-party library. Um, but these are, you know, major vendor stacks. And I do feel like—as much as we talk about open source and stuff—there's a certain level of quality you associate with these major first-party stacks, right, whether it's from Apple or Microsoft or Sun or whoever. That may or may not be true, but hopefully usually is true: that these things are really heavily tested.

Spolsky: There is definitely, yeah, there is definitely... I mean, there's something I've learned over the years, and, you know, I started out with working on the Excel team, um... The developers on that team had a motto, which was "Find the dependencies and eliminate them." You know, they had their own compiler; they would not use untested libraries from other groups at Microsoft even...

Atwood: I love that they had their own compiler. That is so hardcore. I can't even, like, I could not even hang out with those guys... right... that hardcore.

Spolsky: Hey, well, we have our own compiler, man.

Atwood: Yeah...

Spolsky: Let me tell you why they had their own compiler: They had their own compiler because Excel was getting huge, and just compiled 8086 was just too large to fit on floppy disks and to fit in memory. You know, we were really trying to cram things in there. And so they developed a pcode compiler, which basically... you know, it's like bytecode. They called it pcode. This is a very old technique, and it compiled Excel into an imaginary machine, a virtual machine, which was a lot more expressive that an 8086, and had all kinds of additional features, and so the compiled code is about one-third the size, and in a lot of situations this made the performance a lot faster. So, for example, in those days when almost everybody was running programs off of floppy disks, the chance is... Or no, not floppy, but the 3.5-inch, not-so-floppy disks. But the read time on those things is really really slow, so if you could launch your app—if your app was smaller at the time that you read it from disk—it didn't matter if it ran a little bit slower. The whole... the overall experience would be a lot faster. So if you could fit in memory without swapping, then obviously the whole thing would run faster, so it was worth doing this pcode thing for a long time, and about the time of Excel 5.0, the bit flipped on that and it suddenly became... suddenly everybody had hard drives, and nobody really cared about the size of the executable, and it was okay to have about a, I think, a four-megabyte executable instead of a one-megabyte executable, and so they got rid of that pcode back-end. But even then I think they had their own compiler for a while because in order to right really really efficient code, they wanted to be able to control... Now this is a long story.

[9:39]

Atwood: - at the turn of the century. I got these books as a kid and I was totally obsessed with them, because the Great Brain is all about a family. I don't remember the family's name, but there is one central character J.D. who is the Great Brain. Essentially he's always thinking up ways to, essentially social engineering before we had that word in computer circles. Basically getting people to do what you want them to do completely of their own volition. The Great Brain is basically this genius of a kid, who is using all these social engineering exploits to get away with all this crazy stuff. In that family, if they found out the Great Brain was doing this stuff then of course he would get punished, but the ultimate penalty was what they called the silent treatment. The silent treatment meant that nobody would talk to you or acknowledge you for a certain period of time. They would give you food and stuff, but they wouldn't talk at you. It was just stunning in the book, you don't really think about this stuff as a kid (I was like 10 or so, how old I was), how desperate it is as a person, as a social being, when nobody will acknowledge you. How profoundly affecting that is, right? Even the Great Brain, as a smart kid, hated the silent treatment and would do anything he could to avoid getting the silent treatment because it was just such a brutal penalty. I remember Jason Cotke (sp?) was talking about an episode of The Simpsons where these animated statues came to life, and the way they got rid of them was they started chanting "Just Don't Look!"

Spolsky: (laughs)

Atwood: This is led by Lisa Simpson, who says "Just don't look at them and they'll go away!"

Spolsky: (laughs)

Spolsky: Thats right. That was like the Clinton administration's policy on Rwanda.

Atwood: (uncertain chuckle) Well thats the issue of social injustices, which I think is a little bit different.

Atwood: Absolutely.

[23:21]

[57:56]

Spolsky: [We will actually have] I'm in the process of making another movie. Did we talking about this on the podcast yet?

Atwood: Oh really? You sent me a copy of Aardvark'd. I have seen that.

Spolsky: That was our first movie. That was like a slightly serious reality TV show about the interns building Copilot. It was probably better than what on MTV. But the truth is the audience, the Joel on Software audience, really wanted something more technical and a lot of people wrote to me and said "You know, I really wanted to learn how software is developed at Fog Creek", and this movie was just a little bit too light on that content. This is going to be much more hardcore. SO it's really going to be about how software is developed at Fog Creek.

Atwood: Ooh wow... Has this started?

Spolsky: It's about a one year project. We've got the same film maker as we had last time, except I'm basically interviewing everybody at Fog Creek. You know, three or four times, to get footage for this thing. And then we'll splice it all together, and the current plan is to have several different formats of this movie. There'll be a lot of short form little piece, like four minutes, we can put up on YouTube. If you want to hear about source code control, or you want to learn about hiring programmers or interviewing programmers or the phone interview. They'll be these little pieces that you can just download freely on the internet or watch. And then there's a [sort of a] more substantial ninety minute version, which is probably too much to distribute over the internet. But it'll probably distribute in DVD form. Maybe by that time, there'll be a good way to distribute a [you know] ninety minute DVD length thing over the internet. At kind of the high end, and what will hopefully pay for this whole operation, will be a corporate training video. Maybe a five to six hour thing that a team might buy inside a corporation to learn how to do software development a little bit better, and it'll just go into much more depth. There'll be like a whole hour of training on how to conduct interviews for programmers and a whole hour on how to setup your tools and that kind of stuff.

Atwood: Wow. So you're like Howard Stern. You're like the kind of the media now. You have like your movie. You've got your podcasts with me. You've got, you know, your blog, your not-a-blog.

Spolsky: But I mean, you know, that was the original idea behind this podcast remember how I said, like, "There's some people that you can reach in certain channels, and they need to hear you. I don't listen to Eminem's music, blog I saw the movie 8-mile, or miles... whatever it was. Different people, you will reach in different channels. Part of the goal is to do that, and the other part of the goal is: I have an inherent inability to ever give up. And instead of just saying, "You know, we tried to make a movie, and it wasn't as good as it should have been, and so maybe we're not so good at making movies.", we just said "You know, let's learn from the experience. Let's try and make a two point oh, that's just much better."

Atwood: I didn't really hear a lot of negative stuff about it.

Spolsky: You know who liked it? Programmer's girlfriend, wives and families. Because there's nothing about it that's inaccessible to somebody who knows nothing about programming. And it really does convey to the normal person a lot about what a programmer's life is like. Just the sort of feel or being a programmer. The fact that programmers get into these conversations about things. You know that was probably, I thought, one of the highlights of the movie - when we told the interns that their job was to figure out if they could jump across to the ledge of the neighboring building if there was a fire. Could they jump out the window and make it on to the ledge of the neighboring building? And programmers love to have conversations like that, and they immediately leapt up to the white board and started drawing equations from physics and stuff like that, and they tried to work it out. And that's really what our conversations are like, what our lives are like. And to geeks, nobody knows--, if you're a geek you probably don't think anything is strange about this. But to a normal human being... this is very bizarre. That we talk that way, and that we think about these kind of things, and that we then need to solve the problem mathematically, to decide if you can make it to the neighboring ledge before you died.

Atwood: Right... Well that's cool. I'm looking forward to that coming out. Looks like it'll be a while. I definitely recommend people who haven't seen it, check out Aardvark'd. I enjoyed it. I didn't think it lacked anything.

Spolsky: The way to find that is ProjectAardvark.com, and there's link to the movie. ProjectAardvark.com is a blog that the interns kept that summer. It was a couple of years ago.

[1:02:14]

[1:02:50]

Atwood: Two things; we have a wiki, for people who can't listen to this. Where people can contribute transcriptions of our incredibly boring podcasts, and we thank you very much for that. Although I do have one request for the transcriptionists, and the ironic thing is, you're going to transcribe this, which I think is hilarious. When you transcribe, don't write down every time I say 'uh' or pause or 'yeah'. Make me sound awesome, that's my one request for the transcriptionists.

Spolsky: It doesn't have to be word for word. It doesn't necessarily read as well when it's word for word you can leave [out] 'uhms' and 'uhs'.

Atwood: In fact, leave out whole. If you think it reads better a certain way, just make me say whatever makes the transcriptionists sound the most awesome.

Spolsky: And it's wiki, go ahead and edit it.

Atwood: People edit anyway, you're right, it's hilarious. I've been reading the [revisions], it's very funny.

Atwood: The other thing is, if you do contribute to the wiki. Since our beta has been pushed back a week, this will get you in the same day, to the StackOverflow beta. If you want to be in, just email me after you've done a little bit of transcription, one minute or whatever you're comfortable with. If you want to get your question answered on the air, send a less than 90 second recording to podcast@stackoverflow.com, we will put it in the queue and hopefully answer it on the next podcast.

Spolsky: Alright, that's it. Thank you very much, see you next week!

Atwood: See you next week.

[1:04:04]

[Outro]