Spolsky: Today is the day that we did not launch, although we planned to. But then... We'll wait for another week.Atwood: Yeah, well, the good news on that is that we did actually figure out what that problem was.
Spolsky: Oh, oh, I want to hear, I want to hear, I want to hear.Atwood: Eh. So, it was a third party library. Indirectly, I mean. It's the third party library, and our particular use of it. It was Log4Net.
Spolsky: Oh!Atwood: We were logging in such a way that the log.... during the log call was triggering another log call. Which is normally okay, but with the load that we have, eventually they would happen so close together that there's also a lock. So, there's two locks going on there. There's a lock of like disposing of the database stuff that's going on. Then there's lock of like actually writing to a file...
Spolsky: Hm!Atwood: And... Huh... They happen in the opposite order, so it's like a classic deadlock, right. So, you release the lock on the database, then you release the lock on the file. And then the other call was doing in the other order. And they were happening so fast that... it was deadlocking eventually. And it was one of those things that would happen.. like... it was very intermittent, right.
Spolsky: Right.Atwood: So we had to dust-off WinDbg.
Spolsky: How on Earth do you find things like that?Atwood: Well. You bust out Win Debug. One nice feature in Windows 2008, and I think this is in Vista as well. In Task Manager, you can right-click a task and take a dump of it.
Spolsky: Yeah!Atwood: Like right there.
Spolsky: Aha.Atwood: So we took a dump of the W3Service process, and...
Spolsky: Ha ha, take a dump.Atwood: Yeah, I know, any time you do this it's like.. It's like the territory for jokes. It's just...
Spolsky: [giggles]Atwood: [laughs]. And then we loaded up a... Win... Debug.
Spolsky: Windbg! YeahAtwood: ...and then some .Net managed extensions you can, sort of load. You need like a cheat sheet to figure out what the commands are. And then you load the dump, and you load the manage tools. And then you can sort of just investigate all of the threads. You can take the "Show me all the managed threads." And then say "Show me what's the call stack was for that thread." And what we saw was like tons and tons of threads that were all going "Hey, I would like to log something..." And it was like "Hmmmm... [laughs]... Interesting!" Right, you have like 80 threads that all try to write something to the log. So... Right then we kind of knew where the problem was.
And then somebody on Twitter actually volunteered to help us diagnose the dump. So I put it up on our server, and he a.... he nailed... he had a great description of it, like line by line, blow by blow of exactly what was happening. I mean, I'm... I'm competent enough to sort of figure out roughly what was going on, but he really knew this stuff and really helped us out, and I do appreciate that.
Spolsky: That's really awesome.
Spolsky: I'd never .. no ... I never do... I nev... But you know I don't... I don't think I've ever worked on code that is sort of operational in the same way.
Atwood: ah hm.
Spolsky: Eh.. because we definitely eh.. put a lot more ... oh you know, you know what, I did, at Juno we used to have all kinds of logging.
The trouble is that my philosophy has always been that you .. you.. you have a tendency to wanna log everything. But then you just get logs that are, you know, a hundred megabyte per user and you get thirty of them a minute and it can't possibly be analyzed or stored in any reasonable way. So the next thing you have to do is to start culling your logs or just have different levels of debugging, where it's like in high debug mode everything is logged and in low debug mode nothing is logged. And... it's kind of hard to figure out what you really want in a log. You you know you know .. a lot of logs, like I think of the logging that we did in Juno, where people would call with a complaint and you try to figure out where this program is crashing. And obviously a log of the crash, that's easy. Ehm, but then there's some line above the crash which hopefully gives you a lot of information about where it happened. And there's some line you don't see that should have been after that, after the crash, but it never got there 'cause it crashed sometime before there. And essentially what you're doing as you're adding logging, is you're doing binary search, right, where you're sticking in like "well gosh, I got to here and then got to there. But there's an awful lot of code between point A and point B. So let's make an A you know half-way from A to B, log point of some sort". Then you put that in and then you eliminate 50 percent of the possible places to look for your crash.
Um, but I've never really been able to...
Atwood: I mean that, ironically, to troubleshoot this hang, which turned out to be because of logging, we were adding more logging.
Atwood: The joke just writes itself! The joke just writes itself, right...
Spolsky: It does... How many... How many third-party tools do you have... uhh... How many third-party tools are a part of the StackOverflow code base?
Atwood: Well, okay, so... [chuckles] Uh, Dare [pronounces it as the English word "dare"] Obasanjo [pronounces it "oh-bih-san-ho"]... I don't know if I'm pronouncing it correctly.
Spolsky: Okay, "Dare" [pronounces it "daray"]... Obasanjo [pronounces it "oh-bih-san-ja"]... It's "Dare."
Atwood: Is it "Dare"?
Atwood: Really... Okay, I didn't know that. Well, I've learned something. But he had a whole blog entry about how, you know, I had chosen to write my own sanitizer, and that was a very deliberate choice for me...
Spolsky: Mm hmm.
Atwood: ...for a number of reasons that I won't get into. But he was very critical of this, because, of course there were bugs in the sanitizer...
Spolsky: Mm hmm.
Atwood: ...which there were going to be, and to me, it's about, like, it's about your velocity; it's not about where you are; it's about where you're going, and we're gonna fix that stuff, right, and I'm making the sanitizer public as well, so other people can have a sanitizer that's not ten thousand lines of code, and ridiculous, and uh, so there's a philosophy there of building something that's reusable for everyone. Um, but I thought it was ironic, because he was talking about how developers should just pick a third-party library and go with it, and I think obvio... it's a balancing act, because we picked this logging library, right, which kind of caused a problem for us, right, I mean partially it was the way we were using it, but the way it was locking the files was a design issue in terms of the way Log4Net works.
Atwood: So I... I think it's a trade-off. I don't think it's always as clear-cut as "you should always pick a library" or "you should never pick a library," right? I think there's always some in-between there. So, for us, I'm definitely a minimalist—I don't like third-party libraries; I feel like we have a giant third-party library called "Windows," called ".NET"... huh... ASP.NET MVC is technically a third-party library. Um, but these are, you know, major vendor stacks. And I do feel like—as much as we talk about open source and stuff—there's a certain level of quality you associate with these major first-party stacks, right, whether it's from Apple or Microsoft or Sun or whoever. That may or may not be true, but hopefully usually is true: that these things are really heavily tested.
Spolsky: There is definitely, yeah, there is definitely... I mean, there's something I've learned over the years, and, you know, I started out with working on the Excel team, um... The developers on that team had a motto, which was "Find the dependencies and eliminate them." You know, they had their own compiler; they would not use untested libraries from other groups at Microsoft even...
Atwood: I love that they had their own compiler. That is so hardcore. I can't even, like, I could not even hang out with those guys... right... that hardcore.
Spolsky: Hey, well, we have our own compiler, man.
Spolsky: Let me tell you why they had their own compiler: They had their own compiler because Excel was getting huge, and just compiled 8086 was just too large to fit on floppy disks and to fit in memory. You know, we were really trying to cram things in there. And so they developed a p-code compiler, which basically... you know, it's like bytecode. They called it p-code. This is a very old technique, and it compiled Excel into an imaginary machine, a virtual machine, which was a lot more expressive that an 8086, and had all kinds of additional features, and so the compiled code is about one-third the size, and in a lot of situations this made the performance a lot faster. So, for example, in those days when almost everybody was running programs off of floppy disks, the chances... Or no, not floppy, but the 3.5-inch, not-so-floppy disks. But the read time on those things is really really slow, so if you could launch your app—if your app was smaller at the time that you read it from disk—it didn't matter if it ran a little bit slower. The whole... the overall experience would be a lot faster. So if you could fit in memory without swapping, then obviously the whole thing would run faster, so it was worth doing this p-code thing for a long time, and about the time of Excel 5.0, the bit flipped on that and it suddenly became... suddenly everybody had hard drives, and nobody really cared about the size of the executable, and it was okay to have about a, I think, a four-megabyte executable instead of a one-megabyte executable, and so they got rid of that p-code back-end. But even then I think they had their own compiler for a while because in order to write really really efficient code, they wanted to be able to control... oh this is a long story.
Spolsky: So this was this feature that they eventually got added to the regular Microsoft C compiler called base pointers, and I think then they stopped using their own compiler. But their philosophy was really not to trust anybody and to have control over everything so that there's some hope they can get it to work without having an external dependency. You know, I've sort of taken this with me a long ways, and every time I've failed to do that I've tended to regret it. Every time we've put outside technology into Fog Bugz we've regretted it. There's a lot of these excellent components, and they are really great components, that are made by vendors like .Net components and they're widgets like the cool calendar widget dropdown that you put into your web page and all that kind of stuff. And inevitably what I've found is that they are good enough for enterprise code, like internal apps that you're using at the insurance company, and they're just never good enough for the kind of app you want to ship that has to be perfect. Somehow there is something that's not commercial quality about them. You know it's fine if there's 20 people using it, and they're all using it the same way, it allows you to in fifteen seconds put a calendar dropdown into something. But then you'll get to some customer who says, "You know we don't start our week's on Sunday in my country". And you'll say, "Oh", and you'll find out this library doesn't have that feature. As a hypothetical example.
Atwood: I believe that's one advantage of some of the web stuff is because everything is just public facing by default. You don't have sort of this, this internal development ghetto effect. To be fair, any talk of dependencies, we have tons of dependencies, right? It's just a question of what dependencies you wanna take, I mean, jQuery is a dependency, right?
Atwood: We're using the WMD control, that's a dependency.
Atwood: There's these little add-ons for jQuery that...
Spolsky: But you know what, if you found a bug in jQuery, you would just go, edit the source and you would be shipping your own private version of jQuery and problem solved. And it wouldn't be ideal, but at least you wouldn't be screwed.
Atwood: Right ... We have actually done that and let me actually give an example. So the WMD editor has a bug with international keyboards. Like there's no way we would have found this because we don't use international keyboards but obviously some of the people that use StackOverflow do and
Atwood: They were, I think, understandably very annoyed because they couldn't enter like right bracket. Which is like an important key particularly in markdown. That's one of the key, like one of the delimiters you use sometimes.
Spolsky: Yeah, the...
Atwood: You could just view source and if-if you were, you know, motivated enough, you could figure it out, right? There wasn't like an executible that you had to decompile or anything like that. So I feel
Spolsky: [garbled - interrupting] Flash
Atwood: Go ahead
Spolsky: Unless it's Flash all flash. I mean like Yahoo maps is all flash and you can't figure out what they're doing.
Atwood: Oh, right. Right. Right. Right. Right. Well, that gets back into the whole we call it a rectangle problem - the browser where you have this alien rectangle [laughs] that lives in another universe and it pokes a hole into your dimension and then like this crazy stuff comes through and yeah. So it...it...on a related note, let's close out the topic. So the struggling with the deadlock put us back I would say at least four or five days. So in order to have a smooth landing, there's also a couple of features that I really desperately want us to get in before we open to the public like say a captcha for [laughs] I think that's kinda important when we go live. Um. So adding like a week to the schedule really helps us to have a smooth landing. I mean we could launch on the third. I mean we honestly could, but it would be a little desperate. We would be really flailing,
Spolsky: Yeah, no. We're in no rush, we can take another week. I thought that the end of August plan was a little bit ambitious. I think we're both in the same position of really being on the fence, as to whether, I don't want to say on the fence but it's sort of a close call between we want to do the Hollywood launch, going back to last week talking about Aarons Swartz's thing, whether we want to do the Hollywood launch where everyone hits us at once and the world comes to an end versus the Gmail style launch where we just start taking a thousand people a day or give out invites or something to at least have some kind of control over the rate at which people come in.
Spolsky: Yeah. We have quite a finite number of people who listen to our podcast and read out blogs anyway, so they're going to find out.
Spolsky: It's not even public!
Spolsky: These people they're going to the backlash state before we even got to the hype stage. Come on you guys! Backlash comes after the hype, that's why it's called backlash.
Spolsky: For the beta.
Spolsky: Don't even respond. Why are we even talking about them?
Spolsky: blogging-harmful.blogspot.com. Complete waste of time, but you know if
we get people to care about us whether it's positive or negative that means people care about us.
Spolsky: That's the real failure, exactly. If you can't get anyone to care one way or another about what you've done. For example, this website blogging-harmful.blogspot.com is going to disappear without a trace. Even though I promoted it on the podcast it's going to make it all the more painful when nobody...
Atwood: But people who do something like that, they don't want attention or anything, they're not feeding on it at all, they're not interested in things like attention at all. The work is its own reward, it doesn't matter if anyone is looking. Have you seen that thing on the ... I meant to blog about this but the whole concept of just, not looking at things, to basically discourage them. Or that, conversely, looking at things encourages them, like the whole Paris Hilton thing and just talking about these things over and over incessantly actually reinforces the whole trend. There was a series of children's books I don't know if you've heard them, they're called: "The Great Brain". It's set in Utah -
Atwood: - at the turn of the century. I got these books as a kid and I was totally obsessed with them, because the Great Brain is all about a family. I don't remember the family's name, but there is one central character J.D. who is the Great Brain. Essentially he's always thinking up ways to, essentially social engineering before we had that word in computer circles. Basically getting people to do what you want them to do, completely of their own volition. The Great Brain is basically this genius of a kid, who is using all these social engineering exploits to get away with all this crazy stuff. In that family, if they found out the Great Brain was doing this stuff then of course he would get punished, but the ultimate penalty was what they called the silent treatment. The silent treatment meant that nobody would talk to you or acknowledge you for a certain period of time. They would give you food and stuff, but they wouldn't talk at you. It was just stunning in the book, you don't really think about this stuff as a kid (I was like 10 or so), how desperate it is as a person, as a social being, when nobody will acknowledge you. How profoundly affecting that is, right? Even the Great Brain, as a smart kid, hated the silent treatment and would do anything he could to avoid getting the silent treatment because it was just such a brutal penalty. I remember Jason Kottke talking about an episode of The Simpsons where these animated statues came to life, and the way they got rid of them was they started chanting "Just Don't Look!"
Atwood: I also try to copy a lot of things I've seen online that have been successful, like conventions. Let me give you a specific example: PHP-BB, and I'm sure there are other web discussion boards that do this too but PHP-BB is the one I know, has this editing convention. When you post you can edit your own posts, and I noticed that when people are using PHP-BB that right after you post something you'll always notice some goofy mistake that you made, like immediately. This happens to me 9 times out of ten I'll post and think "oh, I should have talked about this" or "I missed that word," so you immediately go in and edit. At a certain threshold these are not treated as real edits, they're treated as just going back in time to pretend that it is the post you originally made. It doesn't kick off the whole auditing trail of you having edited it 50 times. One of the first things we did in StackOverflow is actually implement that. I remember talking to Geoff Dalgas about that, and he's like "why do we have to have this?" I said, "you don't understand, this feature has to be in there on day 1 otherwise we're going to have so many revisions that are just in the first minute or two after posting and are just silly little things that are being corrected."
Spolsky: My feeling is that if the tweak is a little bit subtle and little bit weird, people don't know quite what it is doing. Like that thing you were mentioning earlier about how if you do your own edit within about 5 minutes it doesn't get into the history. See, right then you are doing something that wouldn't be what people would expect. I mean, they might be able to learn that that's the way that it works, but it isn't what they would expect. They would expect that you either see the history, or they don't see the history. It would never occur to them that you'd do something more subtle than that, and therefore, they'd always assume the simple model, and therefore they may have usability problems because they don't understand what the app wants from them. You know what I mean? Basically, usability problems always occur at the intersection, of the user not understanding how the program model works. The program has a model as to how it works, and the user has some understanding as to how the program works, and when those are different that's when you have a usability problem. It may be small and it may be subtle, but that's where you have a usability problem. Your best hope, if you're setting things up and you're saying "Hey I going to give you points if you do X". and everyone does X, and X is something that you want, and you told them that that's what you'll give them points for, and it's obvious, then that's great. But if you're going to do something where you're doing something non-obvious, or a little bit tricky, or you're creating a little bit of a conflict between how they think it is going to work and how it really works, then in all those cases the best you can hope for is that they will accidentally stumble upon doing some behavior that you want them to do. Because of their misunderstanding will cause them to accidentally trip into the particular dark hallway that you want them to go down. That's the best you can do. I think you're always better off striving to make it that people understand what's going on. A lot of times that may mean that you can't have behavior that isn't clearly visible. There's going to be some kind of behavior in there like how you earn a badge, or when wiki edits don't show, there has to be some extreme visibility in the app. It has to explain itself a little bit, so that people understand what it is doing.
Atwood: Right. Normally, I'd totally agree with you. I think this is a little bit of an exception just because again it came out of phpBB, and these other very long well established messaging systems. Were harvesting these ideas from Wikipedia, message boards, and wherever I've been online and I've had a community that I thought really worked. I try to steal those ideas, and fold them into StackOverflow. So I think it is a proven idea that works. It's just a peculiarity of human behavior that you're always going to make mistakes immediately after doing something. So you have that little cusion. It's kind of a special case based on human behavior. And then two, I think we have a community --
Spolsky: What does it hurt to have at least the transaction history shown?
Atwood: Well, because it becomes noise. It's really tiny, simple, edits.
Spolsky: I don't know.
Atwood: Well again, we're not exactly doing... we're a hybrid, right? So, we're in between. So I think we harvest those ideas from different places. I don't think --
Spolsky: Hey I got a question. Heh. I got some questions. Want to listen to some questions?
Spolsky: Sorry, you'll see why I'm laughing in a minute.
Caller: Hi, this is Trey Jackson from Corvallis Oregon. Joel, It's great hearing your ideas. Jeff, It's increasingly frustrating that you don't listen to a single argument that Joel makes during a conversation.
Caller: My question is, assume for a moment that StackOverflow becomes a great success ..
Spolsky: wait, let me just play that again (Laughs)
Caller: .. and gets nothing but good questions and even better answers. How do you intend in keeping folks interested in answering questions when the sheer volume of those questions gets so high. People will google for answers but they'll google for questions ..
Spolsky: (Starts playing question again)
Caller: Hi, this is Trey Jackson ...
Spolsky: I have to play it again and again. (Laughs)
Atwood: (Laughs) That's how much you like it, you wanna play it twice. Wait, I don't understand, why was I not listening to your argument there?
Atwood: I don't get that. I totally was listening, I just think in this one particular case, I mean in the general case I agree the system should be totally discoverable and consistent. But I just disagree that in that particular case It should be. I think it's ..
Spolsky : Well see there's a difference between listening and just accepting what I say and actually that was the reason that I wanted to play that ?? actually is that my problem is usually the opposite, is getting people not to listen to me. For those of you that don't know,for my day job I'm the CEO of a software company. Ahm, and we've got about 20 people and my biggest problem I would say, well actually my biggest problem is dealing with New York city landlords but, my second biggest problem is getting people to not do what I say just because I said so. And it's very hard for them because you know, I'm the CEO and their boss, and I probably have more experience than most of them and when I say do it this way, ah, you know there is a real tendency to try to do it that way. Ah, and it's very easy, it's the easiest thing you can possibly do is just do what you boss says because you have no responsibility over whether it was successful or not because you did what you were told and you can just lean back and just do it that way instead of making the really, really hard decisions yourself. And so I would say my biggest problem is making sure that the developers at Fog Creek understand that that decisions are in their hands, and that If I say "do it this way", I want them to, you know, take that for what it's worth but not to do it blindly and in a lot of cases I just don't have as detailed an involvement or as much intricate knowledge of what's going on. So ahm, I mean to be fair I, you know, I spend a couple of hours a week on StackOverflow or a couple of hours a day messing around on the site. I spend a couple of hours a week talking to Jeff on this podcast, ahm, but Jeff is spending every minute of every day on this and he's much more deeply involved in what's going on and he knows a lot more about how things are and how they work and how they should work, ah, than I do. And, ah, so I may have some general experience to give him but I'm really happy that Jeff makes his own decisions and takes responsibility for them and doesn't just kinda blindly do whatever I tell him because that would be a mess, that would be the person with the least information making the decision. If anything, you know,it takes me years to train a developer at Fog Creek that they should push back when I say stupid things. Ah, or when I suggest, ah, doing things they should make their own decisions, the decision is going to be up to them. I try to remember to always say "look you're going to decide, but here's my opinion". Ah, but it's hard for that to come across as anything other than "you're going to decide, but you're probably going to do it my way because I say so". And ahm, that's really important to me so, ah, this is a critisism that makes no sense .. whatsoever. So there. So Jeff, why don't you answer the part about the umm, the second part. Do you remember that part?
Atwood: Well, right. I want to talk a little bit about the relationship. The parts, I think people aren't really seeing is the discussion that went into building StackOverflow before we even started the podcast. Where Joel has these message boards that he's run for a long time. Joel Knows a thing or two about building a community, rules that make sense, and really big picture stuff.
So a lot of the design that you guys are seeing today in StackOverflow was really vetted by Joel early on. So I think we kind of agree on central tenets but getting down to sort of brass tacks of you know "Do A, Do B or Do C" There's a lot more harmony than you may hear on the call, necessarily. It's been a great relationship. Joel has been very intimately involved. I didn't know what to expect early on, but Joel has been very hands on. It's really been a pleasure. So hopefully you guys are, if not hearing it, then believing me when I say that there are much more harmonious aspects to the relationship as well.
Spolsky: We're always trying to figure out ways to get into big fights on the podcast itself to make it more interesting. But that's just because we are trying to get our listenership up from the current low, two-digit numbers.
Atwood: That's right, your dad needs to know other people are listening to the podcast that his son is on. So the other part of the question was about I think just sustaining interest in the questions over time..
Spolsky: And answering the questions. How are we going to get people to answer questions? It is certainly easy to get people to pose questions, or to show up on Google when we have an answer.
Atwood: Right, well everything that's in the site is in there for a reason. Like the whole reputation system, the way you sort of earn reputation is by answering questions. That's one of the best ways to get reputation. People are actually complaining that when you ask a question, you don't tend to get as much reputation from asking as answering, and I think maybe that's the way it should be. But currently even in the closed beta, people are constantly emailing me amazed that they are getting such great answers and so quickly. So even with the closed sort of petri dish that we have of the beta users, it's already working extraordinarily well. I expect that to scale, like in the Clay Shirky book where he's talking about Wikipedia is just attention divided by a million. You have a million developers with 2 or 3 minutes and we want to have our site be so low friction that they can come on in 2 or 3 minutes and find something they are interested in and that they know about and that they can be authoritative on and then just file a fragment of their knowledge away on our website. No signup , nothing complicated, very very easy. So we think as it scales,you're going to see even more questions get answered. It will be even better than the beta. So the fact that it's already working, to me means that it's definitely going to work and as we scale out it's going to work even better.
Spolsky: Sometimes people even put a question and answer up for their own selfish reasons. You know, it's just a good place to record for their own benefit so they remember how they solved this particular problem.
Atwood: And I totally encourage that. I plan to use the site that way, as a matter of fact.
Spolsky: As a notebook where you put down some little notes as to how you solved a particular problem, no matter how obscure, and other people will come in and improve it.
Atwood: Absolutely. Again the different inspirations for StackOverflow like blogging, Wikipedia, discussion boards, we're trying to hybridize all those things together and this one comes from the blog where I'll blog about something, I'll say "Oh, I thought about this and then I did this" and then I learn so much from the comments I sometimes totally change my solution based on the comments. So even if you come in and post "Okay, here's how I did it" you may still learn a better way to do it from other people seeing it and being able to collaboratively add comments to the bottom of what is effectively a blog post, right? But with no friction, without actually blogging. So I think it totally works and I encourage that.
Spolsky: Yea, we're definitely seeing it working and I think this question is sort of a non-starter. If it's working with 3,000 people, I don't see any reason... You know, we're getting answers with 3,000 people I don't think there's anything different about those people than... there's nothing unusual about those people, they're not... I guess they're early adopters, but... You know there's no reason why they would have a stronger tendency to be willing to go in and play with the site and answer questions just for fun. I mean there's a lot of people that are doing it because it's fun to answer questions.
Atwood: Well, I think too we're harnessing sort of the natural tendencies of the software developer. We talked about this earlier on the call and I didn't actually get to answer it,But one of the reasons that some of the complexities of StackOverflow we can get away with that.. We try to keep it as simple as we can, but we have reputation, there's rules around the reputation... Developers love this crap, right? They spend all day working with rulesets all day long. This is what they know, so they love that crap and the fact that it's a game where you get to show off how much you know. Have you ever met a developer that wasn't dying to show you the best way to do something, like the only true way that anything should ever be done is the way that they do it of course? So we have a community of people that I think are naturally inclined to be very good at this kind of behavior. And we just want to channel it to a positive direction, where everybody's working together, not against each other. That's the only caveat I'll put around that. But I think also our audience helps us succeed because we're building for a very specific audience. We're not building Yahoo Answers, please don't go there, because you'll be very depressed when you leave that site.
Spolsky: You'll be shocked. "How do I tell if my girlfriend got pregnant?"
Atwood: It's just such a big audience and I fell bad because we've actually had a number of people from like WikiAnswers and Yahoo Answers and people that do sites that are similar have actually contacted me and I've been very friendly with them and very open about "Hey, we're just building a site, we're not competing." I want to learn from them,they're going to learn from me.
Spolsky: We're just so much more narrow than Answers.com or, Google Answers is gone, Yahoo Answers, who else has one? Does Ask.com have an answers site?
Atwood: There's tons of these sites out there, there's AnswerBag,
Atwood: There's tons of these sites out there. There's Answerbag... there's tons. You'd be surprised how many Q&A sites there are out there. And I found on Live.com, Microsoft has a site like this. I went to Live.com, I don't even remember why, I guess it's ironic when I go to Live.com: "Oh, look, I'm not on Google, hahaha!", right? But there was a Q&A site embedded in there, you could click on it, it was answers.live.com, I think. And I'd never heard of it, but yeah, go figure, there's lots of these sites.
Spolsky: Yeah, it seems easy to build, but to get the right audience... The other thing that's a little different here is that, to some extent, to be completely honest, we're not building something new from scratch. What we're trying to do is divert a lot of activity that's already happening on these sites behind pay-firewalls or that charge monthly fees, where people are basically contributing answers anyway, even though nobody can see them except for the people who pay for the memberships. So there's a lot of this stuff going on, it's just that it's happening in a kind of inferior way, in our opinion. And it's sort of surprising that developers haven't been able to build something for themselves. The people that get the best tools on the Internet are developers, because they build themselves their own tools. And the thing that Usenet is best for, or was best for when Usenet was popular, is getting answers to your programming questions. And, obviously, the comp.* group of discussion groups on Usenet was always the strongest of everything. So, developers do build things for themselves, but, for some reason, they get good enough and they kind of lose interest. And in the specific area of Q&A, the-state-of-the-art, there were sites that were good enough. But they weren't good enough. Some of those sites that are out there, most of which charge money, have literally millions of programmers contributing to them. So there's a big audience already doing this, and we're just trying to kind of snag some of them to do it in a better, cleaner, more open way, a way that's more visible to Google, a little bit less evil, and so on.
Atwood : Let me give you a specific example: I talked about developers emailing me,surprised that they were getting answers so quickly, that were actually good answers, from the community. This particular person, Sarah Chips, who happens to be a female developer, which is somewhat rare, had emailed me and wanted an invite, and I was ok, if people are heavily motivated, and their emails are interesting enough, I will let them in the Beta, I know I'm going to get a lot of email now, so I'm really regret saying that, but I did let her in, and she actually wrote back a report that said she had posted initially not as herself, because she didn't want to bias the results, 'cause she had found that if you post as a "girl", people tend to pay more attention, and actually answer your stuff in a way that they wouldn't. But she found that even posting anonymously, not as herself, she got a very good answer to her question within like an hour and a half. So, it does, in fact, work. And now she's actually posting as herself, which is nice.
Atwood: But yeah, I'm getting a lot of reports that it's working.
Spolsky: Now everyone's falling all over themselves to answer her question as she asks it. Let's take another question!
Martin: "Hi Jeff and Joel, this is Martin from Spain, I'd like to ask about your overall experience with StackOverflow so far. We all know that you two have been blogging for years now, about best practices, what not to do, or what the common mistakes doing software development are. So the question is, how well did that experience turn out in the real world? Thank you very much, and keep up the good work!"
Spolsky: Thanks Martin!
Atwood: So, what was the question again?
Spolsky: How did your experience help? And it's a big broad question and you can answer it in any way you want.
Atwood: Right, so what did we bring to the table in terms of our years of experience. We're starting a new project. Well, one thing I talked about before is really hand picking the people you work with, I really believe this is one of the most important things you can do in life. really think about who you want to work with and why. And it's all about the people. Every job I've been in, I think that's the one key thing I've learned, is pick a job where you work with people you love. And it doesn't really matter what the job is, on some level, right? Ostensibly it's something you're good at, people would pay you to do, it's all about the people you work with. So obviously working with Joel, god, Joel is legend, really, I'm saying that non ironically, that when I was starting out, I was reading Joel, so being able to work with Joel is, you know, fantastic, it's unbelievable. And being able to actually hand pick people from my previous lives in different jobs, and say "these are the guys that I want to work with", and it just turns out that they were available, I was very very lucky, in that regard, to me, I guess this is the lesson of people, where everyone has different levels of skills, and you can obsess about all the different levels of the process, tooling, and all that stuff, but to me it's such a people thing, programming. And maybe that's why StackOverflow and Coding Horror... It ends up being more about the people that I'm programming with, than the code itself. Which is really strange to me, because you start out with computers in the early 80s, and there's no Internet, there's no modems, barely, it's a very solitary experience. And now it's completely turned around, for me, computing is all about other programmers. It's completely turned around, I don't feel like I'm really programming, and people made fun of me for saying this, and rightfully so, until I produced something that somebody else can look at, I don't feel like I've done anything. If only the compiler has seen what I've done, I have failed. Which is crazy, if you think about pure...
Spolsky: That's true, that's the whole demo, there's just an urge to say "demo", to shout "demo", and everybody comes into your office and see what it is you just built.
Atwood: Right. The whole interaction between people is just what makes it enjoyable, it's the computers plus the people, that's what it's all about. I feel very fortunate to be in the position that I'm in now, where I can be at the confluence of the my two favorite things, it turns out I actually like people! Go figure. I'm not the social outcast that everyone thought I was in high school.
Spolsky: I have a question, this is off topic, but we've been talking about StackOverflow all day. I'm in Boston right now, tomorrow Neil and I are running this two day long Business of Software conference. Richard Stallman is scheduled to speak.
Spolsky: Neil and I have been fighting over who gets to introduce him... Obviously we each want the other person to introduce him (laughs).
Spolsky: What would you say if you had to introduce Richard Stallman to a room full of people who obviously know, who he is, and what he stands for.
Atwood: I think I would go with some of the more; "He's a colorful guy, isn't he?"
Spolsky: Yeh, Red I believe is the color (laughs).
Atwood: Without being a jerk about it, I think you can highlight some things that people may not know about him. That are cool, fun and interesting, and not weird, right? Which is a very fine line.
Spolsky: We're really running out of ideas. I mean, there's the folk dancing. He's in a band I think.
Atwood: Isn't there some online comic [http://geekz.co.uk/lovesraymond] that features him quite a bit, that would be a funny lead in.
Spolsky: Unfortunately I'm scared by a 40 page rider I have to read. To make sure we're not in any way doing anything [bad?].
Atwood: Well, if you have go the careful way, then it can't really be that fun.
Spolsky: You know what would be my intuition, would to say: "When I got to College in 1987, and I had never heard of Richard Stallman, or the Free Software Foundation. The professors told me use Emacs as my text editor, and I launched up Emacs and the GNU manifesto appears. I think it might be one keystroke away from the homepage of the editor. And I read that and I thought; Oh, this is a cool manifesto, I'm into that - I was a founder of a commune in Asia. So the idea of a people's manifesto for free something, sounded great. And I think that the basic thesis, and I'm going to get this completely wrong, was that programmers should work for free in exchange for, maybe, some hardware that's given to them by the hardware companies who need software written for their hardware. And that's the way that I understood it. This extremely, non-subtle, and I know I"m going to get flames up the wazoo, but that's really how I understood it."
And in fact all of the ideas about how free software has something to do with ... well I mean it has a lot to do with freedom and that's one thing ... but all of the ideas that it could have some kind of business benefit or a business structure that would work better are actually post-Stallman. I mean that is open source which Stallman is very much against. Or which he distinguishes very clearly from free software. And... anyway... and I remember thinking you know I kind of disagree with that. I think programmers should get paid and I think it is in programmers' interest to get paid. Now that's the end of that.
Atwood: Right. No, that's fine I mean that's a good intro. I mean I guess I would go more non-traditional. I mean I think he's a very controversial figure and I would sort of use that as a springboard but like I said I don't know if you guys have to be really careful with the intros and things like that. But I mean surely a hugely important figure in computing. I mean I'm... not... you know don't let me take anything away from that. I mean the guy's pretty much a legend, right?
Spolsky: [...] [I don't think there's] been a day since 1987.
Spolsky: ... when I haven't used code that he wrote.
Atwood: Yeah exactly, exactly that's my point. It's like agree with him or disagree with him. Whether you think he's just the weirdest guy in the world or you know somebody that's like a legend that deserves a lot of respect. Wherever you are in that continuum, you have to acknowledge his place in computing history. And very few people get to that position, right? And that's quite an achievement and something that deserves to be respected.
Spolsky: Hear hear...
Atwood: So who else is gonna be at this conference other than...
Spolsky: Some speakers you might have heard of. I'll tell you ... I don't have the full list of speakers. It's kind of weird but half the speakers are people I know and half the speakers are people Neil knows. But Seth Godin who is kind of a management/marketing guru writes about a book a year on a marketing topic. They're awesome. Eric Sink who everybody knows as the founder of SourceGear and one of the original developers of the ... was it called SpyGlass? What was the first browser?
Atwood: Yeah, SpyGlass. And he does ...
Spolsky: So his current company is SourceGear, they have Vault which is a source code control system. And he wrote a book about the business of software which is awesome. And he coined the term "Micro ISV"... this concept that there is a... you know an ISV, a small independent software vendor who's just .... it's Micro because you're making I don't know ten thousand dollars a year writing some software in your spare time. It's really small but it's definitely a lifestyle choice and it's very easy to go from a Micro ISV to making your living writing software, as you do. And so he really coined that term.
We've got Jason Fried from 37signals. We've got... Krug... what's ... remind me of his name... "Don't make me think" ...
Atwood: Oh you got Steve... Steve Krug? Oh man now I'm jealous now I really wanna go.
Spolsky: Well how fast can you get out?
Atwood: That sucks...
Spolsky: Take the red-eye, man...
Atwood: I can't go, I can't go... I just went to Penny Arcade Expo in Seattle so I just had my little trip...
Spolsky: Well next year you'll come but we'll have all different people probably.
Atwood: So, hey... What about people like me who can't go? How can we participate? I mean is there like a recording of the sessions?
Spolsky: Yeah we're gonna record the sessions except that no speaker worth their salt actually allows their recordings to be used like in-whole like as-is.
Atwood: That sucks...
Spolsky: But... you know there is... of Richard Stallman's big talk sure you'd be able to get in Ogg Vorbis format... I'm sure that's right on page one.
Spolsky: It's about a one year project. We've got the same film maker as we had last time, except I'm basically interviewing everybody at Fog Creek. You know, three or four times, to get footage for this thing. And then we'll splice it all together, and the current plan is to have several different formats of this movie. There'll be a lot of short form little piece, like four minutes, we can put up on YouTube. If you want to hear about source code control, or you want to learn about hiring programmers or interviewing programmers or the phone interview. They'll be these little pieces that you can just download freely on the internet or watch. And then there's a [sort of a] more substantial ninety minute version, which is probably too much to distribute over the internet. But it'll probably distribute in DVD form. Maybe by that time, there'll be a good way to distribute a [you know] ninety minute DVD length thing over the internet. At kind of the high end, and what will hopefully pay for this whole operation, will be a corporate training video. Maybe a five to six hour thing that a team might buy inside a corporation to learn how to do software development a little bit better, and it'll just go into much more depth. There'll be like a whole hour of training on how to conduct interviews for programmers and a whole hour on how to setup your tools and that kind of stuff.
So anyway the point of that is that I do actually have the filmmaker here at the conference who helps me film a bunch of stuff. And hopefully we'll be able to have you know maybe just a snippet like one sentence maybe from a speech or from a hallway conversation here or there that will in some way contribute to the overall story that we want to tell with this movie about how software is made.
Atwood: Very cool. Well we should probably cut it off here I think that's the right time...
Spolsky: Alright... let's see do we have any announcements? No we're about...
Atwood: Same stuff. Same stuff.
Spolsky: Ok why don't you do that...
Atwood: Two things; we have a wiki, for people who can't listen to this. Where people can contribute transcriptions of our incredibly boring podcasts, and we thank you very much for that. Although I do have one request for the transcriptionists, and the ironic thing is, you're going to transcribe this, which I think is hilarious. When you transcribe, don't write down every time I say 'uh' or pause or 'yeah'. Make me sound awesome, that's my one request for the transcriptionists.
Spolsky: It doesn't have to be word for word. It doesn't necessarily read as well when it's word for word you can leave [out] 'uhms' and 'uhs'.
Atwood: In fact, leave out whole words. If you think it reads better a certain way, just make me say whatever makes me sound the most awesome.
Spolsky: And it's wiki, go ahead and edit it.
Atwood: People edit anyway, you're right, it's hilarious. I've been reading the [revisions], it's very funny.
Atwood: The other thing is, if you do contribute to the wiki. Since our beta has been pushed back a week, this will get you in the same day, to the StackOverflow beta. If you want to be in, just email me after you've done a little bit of transcription, one minute or whatever you're comfortable with. If you want to get your question answered on the air, send a less than 90 second recording to email@example.com, we will put it in the queue and hopefully answer it on the next podcast.
Spolsky: Alright, that's it. Thank you very much, see you next week!
Atwood: See you next week.