The StackOverflow Podcast

Ads, intro

[1:00]

Spolsky: Let's talk about, umm, that thing with the thing with uh, oh what did we want to talk about? Oh! Voting, voting on questions. The StackOverflow UI...

Atwood: Right.

Spolsky: The StackOverflow UI issue.

Atwood: Right.

Spolsky: So...

Atwood: Do you want to give contexts? Because, you know,  not everyone can see the site yet. You wanna explain the situation?

Spolsky: Yeah so, uh, so uh StackOverflow lets you, it's a Q&A site, people ask questions and then other people produce answers. Sometimes other people [produce answers]. And, err, you can vote on the questions and you can vote on the answers. Umm now voting on the answers everybody seems to have figured out that's pretty obvious, you vote on the best answer. And that seems to be like already working, like even on an obscure question with only three votes total, the good answer has the more votes and the bad answer has, you know, negative votes, and the idea of what a good answer is, and what a correct answer is, uh, seems to be pretty canonical, but, but how do you vote on questions? Like what should be the rule for when to vote a question up?

Atwood: Right. Well, I agree there's some confusion around the feature and one thing people have proposed that I don't entirely agree with is; they've said that votes on the questions should simply propagate up to the, to the, [correcting himself] votes on the answers should propagate up to the questions.

Spolsky: Oh.

Atwood: So every time you vote on an answer, every time you vote it up, that votes the question up, and every time you vote the answer down, that votes the question down as well.

Spolsky: That umm, that might work. No, I'll tell you when that doesn't work, uh, that doesn't work when, when there's a question - you're like "God, yes, I'd really like to know the answer to this so I'm gonna vote it up, 'cos this is a good question, I'd love to know why this is". But there's no answer there...

Atwood: Right.

Spolsky: so you want to be able to, to ...

Atwood: Right.

Spolsky: Well the real question is, what is, uh you know I was thinking a little about this last weekend and I was thinking - what is the effect of voting on those questions, where does it show up; well, um, the main thing it's going to determine I think is the default homepage. And I think the ideal, what we want the default homepage to do is... give a random person that just arrives, um, a list of interesting questions that changes frequently. So all the questions on there are interesting, uh, whatever that means, because people have said "yes I agree that this is interesting" and they are also changing pretty frequently so you're tempted to keep coming back, so if you're just bored sitting around at your computer and you're like "yeah, I'd like to learn something new about programming", you can go to StackOverflow and see a bunch of new things and two hours later when you're bored again you can go back to StackOverflow and see a bunch of new things. And the next day it'll be all new so there's something of a Reddit or a Digg-like homepage, uh, where, uh, interesting things are constantly being voted on and that's what gets them on the homepage.

Atwood: Right. So one thing people have been clamoring for, and rightfully so, is - they want a way to sort of keep track of what things they looked at, and you pointed out [in] an email to me, 'cos we were, the way this all came up was; we were talking about algorithms to determine what the hot items are; the items people are most interested in right now, and one side-effect of that was thinking about voting on questions and what does it mean. Uh, because it has come up a number of times, and you said that maybe what we should do, rather than voting on the questions is just have a star on the question where you just sort of favorite it. And then it's more like del.icio.us where you just say ok I like this question, this is an interesting question so I'm just gonna put it in my favorites and I can go back to it later, and one side-effect of that is you build this, you know, breadcrumb trail to the things you're actually interested in. Now, I kinda like the system the way it is now, I can see where people are coming from with wanting this access of just tagging things as favorites, I get that, but I think voting on a question is similar to, but not exactly the same thing as, marking something as a favorite. I think it's really closely related, it's not exactly the same thing, I mean you could see yourself voting something up that you're like "Oh I don't know anything about this, but wow, that's a really good question". You know, the person who asks this question really did a lot of research, and has some interesting detail and wrote really well, and it's just a really good question, right, I can identify that even [if] I have no knowledge of the topic really being discussed. [You could] say "This is a really good question, this guy clearly put time in to that, that deserves a vote up, that doesn't mean it's my favorite question, right [laughter]...

[5:28]

Spolsky: Yeah.

Atwood: Uh, but a lot of the time it will, I think things I tend to vote up, are things I a. understand and know about and would click on it anyway, right, like "Oh, that's an interesting topic to me" so I click on it, um...

Spolsky: Mm hm.

Atwood: And I think another point of distinction here, we don't allow people to vote on things until you click through to the actual full detail. And I think that's actually a really big mental shift you have to make from things like Digg and Reddit, 'cos that's just one layer of our system, and for those guys you're browsing a list of topics and you're just mindlessly going click "hate hate it, love it, love it, hate it, hate it, love it"...

Spolsky: Yeah.

Atwood: You know, you're just, it's just totally not the same thing 'cos in our initial, like early in the beta before we went public, we had that in, and Jared said to me - he's like "I really disagree with this, I really don't think we should be letting people vote on things this way...

Spolsky: Until they at least part[icipate]...

Atwood: [Interrupting] Yeah, until they, yeah what are you voting on?! You're voting on the title of the article?! Did you read it? Do you know... [laughter] I mean, it makes total sense and when Jared said that I was like "you're absolutely right, we should not allow this", because until you've clicked through you don't even know enough, you're not equipped to vote, right, you know it's like voting just blindly on, you know, just a ballot just checking boxes that's based on somebody's name...

Spolsky: Mm hm.

Atwood: Uh, so there's a lot of subtlety to these decisions, and I'm not saying we're making all the right decisions, but I do understand the need to have that breadcrumb trail of stuff you've voted on. So one thing we currently are working on, and will have in the next few days, is a private page, 'cos voting is another one of those things where we can't really tell other people what their votes are because, it would just get really weird, do you know what I mean? [laughter]

Spolsky: Mm hm.

Atwood: Like, if people knew you had voted their stuff down [laughter], or I had voted their stuff down they would probably get pissed right?

Spolsky: Do you even, do you even track that?

Atwood: Oh yeah we track it all, we know, I mean I don't look at it, I don't care frankly, but, oh yeah we track it all.

Spolsky: So it's not just like a, you're not just incrementing a number?...

Atwood: No, no no there's data of actual...

Spolsky: You're actually recording a vote of [garbled]

Atwood: That's right. And uh it's funny, Geoff Dalgas, we were working on this the other day [laughter], and of course we're constantly changing all the rules and redefining everything that we're doing and Geoff remarked that in uh, in the Skype window he's like "man", he's like "I'm glad we don't make voting machines" [laughter], it would just be a disaster right? It's like, our current system, because we're currently recalibrating a bunch of things and I think we're about to piss off a bunch of our audience 'cos we're going to put in a whole bunch of reputation caps which they're not going to like! And I've also been clearing out a lot of the old meta-StackOverflow discussion that is a) old and b) resulted in a lot of reputation gain for, like, topics that don't really have much to do with actual programming questions.

Spolsky: Well that should, um, yeah, I mean, if the system is designed right, it shouldn't hurt to have, I mean the system was designed so old things could stick around, even when they weren't relevant any more...

Atwood: Well, well [Garbled]

Spolsky: And there was a reason to, uh...

Atwood: ...stuff in the beta though, that's not even there...

Spolsky: But that's the whole point, StackOverflow is supposed to work for like, like this is the question you were already asked on that podcast, what was the other podcast called? Hard?...

Atwood: Herding Code. Herding Code [emphasis].

Spolsky: Herding, Code.

Atwood: Yes.

Spolsky: Herd. Herding.

Atwood: [laughter]

Spolsky: [Garbled: giving?] Is it, uh, is it a sheep? A metaphor to do with sheep?

Atwood: Well it's like herding, herding cats right? The whole herding cats metaphor. It's a pun, it's punny, you know herding cats, herding code, you know.

Spolsky: Ok. So, uh, [laughter]...

Atwood: I see you do not approve but uh, let's proceed [laughter].

Spolsky: [laughter], I'm sorry. What, what were you ask[ed]... you were asked something about, like, y'know the problem, and this was something we recognized when we built this, the problem with a lot of those websites is you go, y'know even to the official Microsoft discussion forum on some particular technology, and you ask some particular question, and you find a bunch of people talking about the beta of three versions ago, and the information that's there is, is no longer, uh, correct or up-to-date and...

Atwood: Right.

Spolsky: What's worse is that particular question now has, like, the PageRank, so that's locked at the top of the Google results, you're stuck...

Atwood: Right.

Spolsky: With this old rotten question, that's much higher in the Google results than maybe somewhere somebody else on the internet who actually has up-to-date information. Just because Google tends to give, uh, a lot of, uh, just both the way PageRank works and the way Google works in general, you know the old, the longer something's been there the more cred it has, the higher it's going to show up in the ranks, so, uh, um, so that's the reason we built in this wiki capability, we wanted these questions to stay up-to-date, and if someone went in to the question and found it was old, they could go in and say "Well", you know, they could at least edit it to reflect that it was no longer the case, and hopefully, uh, put in the new data and that's what I'm kinda hoping, uh, will happen, it'll certainly happen with the very canonical questions, the ones that are, uh, very very commonly looked up - how do I iterate over row sets in C#, that's gonna be, that's gonna come up all the time, people are always gonna ask it, people are always gonna look at the answer, and if the way you do those changes, um, people will edit the answers I think. So, um, given...

[10:37]

Atwood: I agree with all of that.

Spolsky: Yeh - So why do we need to delete old irrelevant StackOverflow questions?

Atwood: I think there's a little bit of a special case like as you're launching a site. Right. Because we're like desperately busy actually writing the site so, but then we also have to participate in the site and see what's happening and gauge what impact the changes are making and make sure people aren't going to kill eachother... and make sure things are working the way they're supposed to. So we sort of dip our toes in the water but i think in this early phase we weren't providing a lot of guidance about like what is this site i didn't have a FAQ together of any significance so they just had to figure it out. They had to like guess and try and that's great and that was creating a huge bunch of questions about StackOverflow and for a long time StackOverflow was the second largest topic of discussion at the site. This has changed and also I've cleaned up some of the old stuff and you would expect that to change over time... but some of those old posts were just you know basically after birth... not really useful... I'm using that in a very intentional way... I do not view those as useful things... a lot of the other questions that are great questions and even the ones that I don't really care for sometimes - are too discussy - I'm not going to go in a delete them - I'm not going to be that guy - that goes in and just deletes other people's crap - but I think I have to be on some sort of level because you've just gotta cleanup... the act of creating the site creates some unnecessary complications that you should just remove later - like warts - and after - if people want to create new threads on this - that's fine - within a certain limit right. I'm looking at the overall ratio of questions on the site and trying make sure that there's not too much naval gazing - because I have had - I've actually gotten to the point where I don't like it any more - honestly - I think there's too much discussion of the site - almost ad nauseam.

[12:30]

Spolsky: It's um, it'll, uh, first of all it will settle down.  But secondly, you'll have tools.  Like you'll, I don't think everybody's always going to be looking at all feeds.  I think they're going to be looking at technologies that they care about and they'll look at the home page, and the home page will inevitably have one thing about StackOverflow every day from some newbie but they'll just get voted down if it's not interesting to the larger part of the audience.  I mean, that's what I saw...

Atwood: The voting is working, the voting is working spectacularly. I mean, I agree there is some question about how effective the question voting is but I think even the question voting it's a less strong effect, but it still works. You still see good questions get voted up.

Spolsky: So, one of the things I like about Google is that they have this almost fanatic devotion to not, no special cases, like they never go in an just remove a site from the search results. They, they, or tweak the search results in any way, they always try to change the algorithm or do something algorithmic. And uh, it got so bad, there was this incident, do you remember this incident? With the search term for "Jew"?

Atwood: Vaguely

Spolsky: It turns out that no one uses the word "Jew" very much. They'll say "Jewish" or "Judaism" or something like that and actually using the word "Jew" is kind of the hallmark of an anti-semite and so if you searched for "Jew" on Google for a long time, until recently, it's been replaced by the Wikipedia article but for the longest time you got "Jew Watch News" which is this huge anti-semitic site. You get, um, a lot of anti-zionism and all that kind of stuff and, um, that was just what was naturally happening and that was not because the world is anti-semitic, it was just because the word is not used that way any more. You know, people always say, "jewish person", or something, they don't say "the jew" and, um, and so, Google was absolutely unwilling to tweak any of their search results no matter how offensive, um, the appearance was there was sort of a lot of conspiracy theories and paranoia among jewish groups that Google was somehow deliberately being anti-semitic or something by high-lighting these anti-semitic sites when you search for "Jew" and um, what they finally did was create a web page explaining why those results are the way they are and they bought an ad on their own site explaining it. So there is an ad, if you search for "Jew" on Google now, you'll see an ad that Google themselves has bought that explains why the search results are what they are and why they're not going to do anything about it. And uh, so I that's something I thought was sort of interesting is like, they've always had this uh, fanatic goal of not doing anything, not really special-casing themselves or just not being willing to have any exceptions but thinking, "well this is a sign that there must be something wrong with the algorithm and I'll go back and tweak the algorithm". So in that spirit, if we're getting a lot of discussion about StackOverflow, maybe there is something about the algorithm we can do, like if it's highly repetitive, then maybe the algorithm isn't doing a good-enough job of directing people to previously asked questions.

[15:48]

Atwood: Right, and I think there's two usage models here, and I think we're only really seeing one because it's a private beta and, you know, the nature of the audience is that they're highly interactive and very interested, and I think you're going to have vast hordes of sort of vaguely interested people that are going to hit the site, and those are really supposed to be a big big part of our audience. These are the people looking for drive-by answers.

Spolsky: The beta does not in any way solve the people who come to the site, because the beta is invisible to Google, so our most common use case which is gonna be somebody types a question into Google and the answer happens to be on StackOverflow is not even being tested in the beta. We don't even have that kind of audience.

Atwood: That's right. I think that sometimes people forget that thats really ... I don't understate the importance of the people that are highly engaged with the site and want to do it, but thats kinda like ... what is the term they use in the movies? "How will it play in Peoria?" The mainstream audience you're going to get is not really the same as the elite developers that you have attached now. And although people have complained about the quality of the questions, you have yet to see what will happen when we open this up to the real world.

Spolsky: Yeah.

Atwood: Like the offensive flag on the last podcast that we talked about. People were confused about that. I don't think they'll be confused at all what thats for once we open it up to the public.

Spolsky: [Laughter] You mean because there will be truly offensive things will show up very quickly.

Atwood: Oh, Yes.

Spolsky: Is there a tag for "Jew" yet? [Laughter] No, we don't have that.

Atwood: Nice.

Spolsky: Because I personally would be very offended.

Atwood: Way to tie those together.

[17:32]

Atwood: I want to come back to one thing I talked about and I realise I'm meandering and I apologize, there's a lot of stuff here. We do want to show peoples vote history because I feel that there's a strong overlap between the things you've voted up and sort of your trail on the site of things that you're kind of interested in like .. people were complaining its like well if I wanna find something that I've looked at I have to post an answer there and then it shows up on my user page as you know something I've answered but know once this private tag becomes visible we can't show it everybody we can only show it to you because it actually shows your actual voting history you can use it as a trail of breadcrumbs say oh these are the last 50 things I voted up and then you can just walk the list and go back to those. So it is kind of a de facto kind of favorites... as there is a relationship there..

Spolsky: Oh I see here on the... When I look at my profile page it says zero votes is that where thats gonna go? ..[garbled]

Atwood: Oh you haven't voted on anything?

Spolsky: Yes, no I vote on things all the time.

Atwood: Oh well we had lets see [chuckle] let me see

Spolsky: [laughs] OK we still got some kinks to work out I hadn't even noticed.

Atwood: No no no actually I know what happened we actually had, heres some full disclosure. So this is a site for programmers so we have a lot of programmers actually attacking the site and one very determined group actually did make a successful cross-site scripting attack and they were actually able to log in as me and you unfortunately.

Spolsky: Oh

Atwood: Yeah well I learned a lot about this actually I'm going to write a blog entry about this but one of the unfortunate things is that there is this thing called a er, well cookies those aren't complicated. But if you can track someone's cookie if you can basically inject script into the page that takes the cookies via JavaScript and just dumps them out to another website

Spolsky: Yep

Atwood: Then that person essentially becomes you

Spolsky: yeah

[19:27]

Atwood: Right. I mean unless you're... Even with a perfectly encrypted connection this is absolutely possible.  This is why cross-site scripting is so pernicious and so dangerous.  And it actually did happen, and I'm actually kind of annoyed at those guys because I've been tolerant of their activities because usually they tell me, and warn me.  But they've really been skirting the line between just sort of doing things and not telling us.  Like we had to sort of, see the logs, and sort of figure out this was happening.  And, uhh.. yeah.  There may be some penalties.

Spolsky: They should be reporting things as bugs instead of just exploiting them. I mean, we're happy that our beta testers are discovering these things while it's still a beta.

Atwood: Right.

Spolsky: But it would be...

Atwood: Well, I think these particular two are just showing off.  I mean, even talking about it is bad because they want attention.  I mean, it ultimately kind of helps the site but, like he mentioned that he actually tried to delete your account and I think the votes in fact did get deleted and I apologize.

Spolsky: Okay.

Atwood: This is the kind of thing we want to find out you know, before we go public.

Spolsky: Yeah.

Atwood: So, yeah.

Spolsky: Hey, what did you think about, here's something that people who are not paying attention to StackOverflow can relate to...

[silence]

Spolsky: [laughs] I'm sorry. I just got an email from Barack Obama that says, "Did you see Michelle?". And I'm like, "Oh dude. [laughs] Please tell me you didn't just email your whole mailing list because you don't know where your wife is."

[both laugh]

I'm sorry, he needs some email training.  Okay, where was I?  Umm, now I've lost track.  What were we talking about?  Oh the beta..

Atwood: We were talking about...

Spolsky: Did you see that article that Aaron Swartz wrote about how to go public? Like not the Hollywood Big Bang launch, but like a long, quiet, steadily growing.. Did you see that?

Atwood: I don't think I've seen that.

Spolsky: That was pretty good. I'm going to have to find the link. It showed up on Hacker News. It's just on Aaron Swartz's blog. Let's see...

Atwood: Doesn't he do politics now?

Spolsky: Yeah, he does all kind of stuff.

Atwood: Did you know his page is PageRank 9? Did you know that?

Spolsky: Nah, it's four.  And umm..

Atwood: [laughs]

Spolsky: I'm looking at it right now, it's four.

Atwood: Okay.

[21:48]

Spolsky: The article is called "How to Launch Software" and he is sort of comparing the Hollywood launch which is is kind of what Apple does really. It's absolute secret and then they throw... impose it on the entire world. That's actually what Cuil did and this was a disaster. They just weren't ready. And the opposite is sort of the quite like the GMail launch where it's invite only. And then you can  send invites to your friends but they very much controlling the speed in which it rolls out.

And the benefit is of course that you get to scale gently rather than suddenly.

Atwood: Right.

Spolsky: Let me tell you. I'm just gone read from his blog post something quickly.
"What happens when software developers try the Hollywood Launch, and I've seen this many times, is that users indeed do flood to your site on launch day but first they bring the site down from the load. You scramble to get it back up and succeed by coding like a mad man, only to find they discover some big bug that you never quite noticed before, which makes the whole thing look like embarrassing hackwork. So you're desperately rushing to fix the bug before the traffic dies down, rush-patching things and restarting the server when you bring the site down for everyone because there was a syntax error in your patch that keeps the server from coming back up.
You fix it while cursing yourself madly. Finally everything seems to work. You take a breath and decide to see what people are saying about you on the Web, only to discover that everyone misunderstood what your product does because your front page wasn't clear enough. Now they all think it's stupid and wonder aloud how you even know how to breathe. So you reply in all the comment threads and fix your front page to ensure no one could possibly misunderstand what it is you're doing just in time to find that all the traffic is gone." So this is exactly what happened to Cuil [laughs].
And I've seen it actually a few of times. It is kind of interesting point like if you get publicity and you get everybody looking at your website kind of suddenly all at once...
you often get a very very large number of people who have a worse impression of you than you deserve.

Atwood: Well there some conversations among the many threads of meta conversations on the site. One that I actually thought was someone insightful was the idea that, and I've often thought this. This is like when you have something good you actually don't want a lot of people to know about it.
I mean the fact that [sort of?] only the 'cool people' know about it for a lack of phrasing it.

Spolsky: yeah

Atwood: Is actually a kind of a plus. And you know. Having a sort of the dogs of digg sort of run over and trash your place doesn't necessary help things.

Spolsky: yeah

Atwood: On the other hand I'm torn because I definitely like you know tight knit communities and I've talk about that many times on this podcast the value of those tight knit communities.
But on the other hand I want to be inclusive to were anyone can participate like the whole Wikipedia aspect to it. I feel like how do you balance that.
I mean. How do you have. I mean I guess you just not publicize it. Which we could certainly do. I mean I could certainly not write about it. You could not write about it .
It's getting plenty traction without us ever even talking about it.

Spolsky: yeah

Atwood: So it's an option I guess.

[25:00]

Spolsky: Well I like you am very very torn. Because first of all. I think that. A lot of the power of this starts  to kick in ...there's two things where the power of StackOverflow really starts to kick in. Number 1 when Google can search it, and therefore it starts to provide a service to people who are searching for obscure technical questions using Google, and they don't know about the site... and ah Number 2 when the audience reaches critical mass so that there is a reasonably large number of people who are writing classic VB 6.0 XML code. ah and or something just obscure and very very nichey. And when you hit a critical mass with those people. All of the sudden the value goes way up. Instead of having just the top questions you actually start to get the kind of obscure questions. And get those answered. And then the value goes up dramatically. Because we really are... you know there's this whole theory of the long tail... and Wikipedia is kind of long-tail-ish... there's this article on the comic book character "Little Dot".  But even Wikipedia doesn't purport to have an answer ... an article... about some obscure ah... technical question "How to programmatically insert RTF text into a document in (Microsoft) Word from a macro without overwriting the users clipboard" for example. Which is the question that I answered yesterday.

Atwood: Right.

Spolsky: That's so obscure and so narrow that I don't think anybody who participates in Wikipedia would expect that there would be a Wikipedia article about that and so we're going after an even narrower longer sliver of the narrow long tail. And to really make that work you need the scaling and you need the mass and you need everybody.

Atwood: Yup... I agree. And I've had the same concerns about the "hollywood launch". For one thing I honestly don't even think it's even necessary. I think that people that need to know about StackOverflow kinda know about it... and the people who should know about it will because of the whole "network effect" of the thing were doing. So i guess maybe yeah well have some comments on this podcast that'll maybe give us some advice on the directions that we think we should go. One thing I don't want to do is have like an invite base system ?like google? remember the early days of gmail?

Spolsky: Yup.

Atwood: When you could mail like 100 people and those people could join.

Spolsky: I think you got like 5... you got 5 invites.

Atwood: Well you got more over time.

Spolsky: there was a knob that they turned to adjust it based on the amount of physical hardware that they had and how much they could support ... and they would just increase the number... they would give everybody more invites if they had extra hardware and they would take them away if they didn't.

[27:35]
Atwood: Yes

Spolsky: But, okay, well here's the thing. The truth is, it's not really in our hands right.Because, we're going to blog about it. We have the audience that we have, and that audience already knows about it and a lot of them are listening to this podcast. And they're just hearing about it through our website, and they're going to find about it. And then there's the second tier, next larger audience, which is you know, Scoble already mentioned it. It's likely to get looked by, even if we don't promote it to them, by people like I dunno maybe Tech Crunch, or you know that kind of thing. It will show up on Techmeme for a minute. And when that does finally launch, a lot of tech bloggers will come in and looked at it, and mention it. And that's sort of a medium level of publicity.

But then there's a thing which we probably don't really want to happen which is, front page New York Times [laughs]. That would probably be ah, bad, for us early on. First of all, the traffic would overwhelm us. Secondly, those people wouldn't really know what they were looking at necessarily, or how it's suppose to work. And we would just have this, phenomena of a million newbies running around.

Atwood: Yeah, I think the worst thing that we could do at this point is for you and I to both to both talk about it at the same time in a very public way.

Spolsky: But, we don't have.. I don't think that would be.. I mean.. Our audiences are perfectly a good developer community. They'll be fine. The trouble is, if it kinda goes broader than that for some reason.

Atwood: Right.

Spolsky: I goes to the... if it starts out and people who don't really know how to use, I don't want to say, "don't know how to use the Internet". But, the audience of people that read Coding Horror and the audience of people that read Joel on Software are already fairly elite in the programmers. Because they're the kind of people who are trying to read things in order to better themselves as programmers. And that's already, you know, 5% to 10% of practising programmers. It's not the vast masses of Java monkeys, who were formally VB monkeys, who were formally COBOL monkeys who are just doing you know, large swaths of extremely boring stuff internally somewhere. Ah, who have I not yet offended?

[both laugh]

Atwood: Well I think the Jews. You've offended the Jews for sure.

Spolsky: [laughs]

Atwood: Big trouble, buddy.

Spolsky: I hope. Sorry. Don't bother writing in, I'll just commit suicide. I apologise in advance.

[30:07]

Atwood: Let's talk about the schedule...

Spolsky: Yeah

Atwood: So we did decide... so the only show stopper that we had was the deadlocks; and I actually blogged about that. It turns out for our particular, actually for most web apps, you have this load scenario which is basically like 99% reads and 1% writes. [laughs]

[30:58]

Atwood: And it turns out that SQL server is kind of crappy for that kind of load scenario out of the box, um, But they added this thing called recommitted snapshot which is exactly what it's designed to accommodate - those kind of loads.

Spolsky: Is that the same as row-level locking?

Atwood: Ah, I get so confused, there are so many different features in SQL Server and,honestly, I can't even keep track of them all. The one thing I do know is that every time you read about this thing called read committed snapshot. So, that's exactly what it's designed to accommodate: Is to have unblocking reads, most of the time, like in except for extreme scenarios.

Because, it turns out the locking model in SQL Server out of the box is very conservative.Like, it likes to lock stuff, really longer than it probably should. If you don't, you know your not.. You don't have a bank account you're keeping track of here. It's just some posts on our stupid website that we're creating. So, if it's out of date, that's better than this giant blaring deadlock error.

Spolsky: So basically what we're doing is declaring to SQL Server: "I don't care if the data is slightly out of date, as long as it's consistent. As long as it was correct as some time. If somebody is changing it right now, don't wait for them, I'll take the old data please."

Atwood: Yes, for the most part. I mean, although you can still lock. This is not lock free by any means. This is not.. the lowest level is what they call the dirty reads, which is the no lock stuff. And you know, that's where you get into slippery slope of, that is kinda sketchy. Because you're basically, disavowing all locking at that point. [laugh] So we're not disavowing locking, we are in a consistent state. It just may be out of date at any given time. But I'm really glad that we've resolved that and I've got to thank Geoff Dalgas for really going the extra mile on helping us figure that one out. But, that was the only show-stopper that we had. The rest of it is pure scaling stuff, right. Which we can do. We can figure that stuff out. It's not an unknown.

But for the record, in terms of scaling, I am inviting more and more people in. Sort of, now I'm doing 300 people per day. Trying to get the load up as much as I can. And it still actually looks really good. We have an 8 CPU box with 4 GB of memory and it's humming right along. We don't.. we're not even remotely close to pegging the CPU. The caches are all working. We're not missing any major indexes. I just ran the Database Engine Tuning Advisor right before you called and, have you ever run that thing? It's kinda funny.

Spolsky: Yeah, regularly actually. Yeah.

Atwood: Yeah. But it comes up with the craziest indexes. "Oh yeah, you've got to index these 10 fields! And these 12 fields!"

Spolsky: And you can either try to figure it out or just say "Okay, do it." [laugh].

[33:00]

Atwood: Yeah, exactly. So my strategy with that was, I picked.. it found one index for us that was 19% of our workload, which was really high. And the rest were like, 1%, 2%, it's like, "I don't care". So I created that one particular index, which ironically created some deadlocks while I was creating the index but that's acceptable because I knew that it was going to happen. Of course then people started emailing me, "Your deadlocks are back!"and I'm like, "No no no, I was creating an index. It's okay, no need to panic." [laugh] So yeah, stuff like that.

We actually have yet to implement any kind of caching on our end. In other words, every time you hit a page on StackOverflow you're actually querying the database every time. This is intentional, okay. Because we're trying to optimise the normal state of the system. So that, in a normal page request we're not killing the database, right? We want to be as optimal as we can with no caching. And then we're going to make it even better by adding caching on top of that. That's over the next few days.

And, have we even said the date? We were tentatively thinking September 3rd. Right Joel? Is that what we thought?

Spolsky: Umm.. what day is that? Like right after Labor Day weekend, yeah.

Atwood: 'Cause we figured nothing is going to happen on Labor Day weekend.

Spolsky: Yeah. I was actually thinking, yeah the 3rd. The 2nd and 3rd.

Atwood: Yeah, and I'm comfortable with that based on the features that we have, the fact that we've resolved that annoying deadlock issue, and my comfort level with our current level of load. Now, we could get load like that's so crazy, that who knows [laugh]. But, our ace in the hole is we're going to go back to this caching thing and actually start caching stuff and hopefully that'll buy us some time.

Spolsky: Yeah, you also could certainly add another box and move SQL Server to its own box.

Atwood: I'm a little worried about that because I feel like the advantage to having SQL Server on the same box is that there's no network communication time. It's like a TCP/IP socket to the same machine. Right? There's no.. So I think by the time you...

Spolsky: No, but.. It helps, it helps. Because with gigabit Ethernet, it's faster than the hard drive speed anyway. So..

Atwood: But it's in memory, it's not really touching the hard drive for the most part.

Spolsky: I'm saying that if you have two machines that are hooked up and one of them is SQL Server and one of them is the web server and they are hooked up with a dedicated gigabit Ethernet. All your good servers will have two Ethernet cards. So you can just have a single Ethernet card that all it's doing is letting you talk to your SQL Server. And the speed of that network connection, will not be saturated until you're pumping out data faster than you're reading it from the hard drive, which is impossible.

Atwood: But still there's more latency going over the network than there is going in memory on the same machine. That was my concern, was the latency across..

Spolsky: It's negligible. Yeah, it's negligible. We do that everywhere and everybody says that it's worth it.

Atwood: Okay. That's cool. We do have that path...

Spolsky: I mean, don't forget that, what happens is: I mean you have a lot of CPUs. So..

Atwood: Oh no, we have a ridiculous amount of CPU power, it's absolutely not the bottleneck. Most of our CPU time actually is SQL Server any way.

Spolsky: The other thing, I mean there's going to be a point where you don't want to be down that much and you're going to want to have two web front ends that you can swap in and out and just one SQL Server back end.

Atwood: Right.

Spolsky: That's sort of the standard, you know, slightly higher tolerance system. Where you have a front end load balancer, two web servers. You can bring one down to update the code. And then bring the other one down to update the code. And you don't, for just regular daily, "Oh, I have new code", you don't have to bring the server down.

Atwood: Yeah, this is going to be quite.. I think you were right to be concerned and I was also concerned that, you're right. Once this goes live there could be a really high level of load and I could spend a lot of time not necessarily programming, but doing sys admin type stuff. Which I don't mind actually, I view that as a part of the whole ecosystem of understanding.

Spolsky: Hmm hmm.

Atwood: You know, if you write good code, you understand like how to deploy that code.

Spolsky: Yeah

Atwood: All these things we just talked about are..

Spolsky: It's much more fun when you get to work on both things at once instead of just having to stick to coding or sysadmining. But, what I'm really.. one of my fears actually is we'll suddenly finding ourselves in very reactive mode. Where we spend literally all of our time just putting out the latest fire that is caused by the sheer size of the site and we don't have enough time to do features that we might want to do.

Atwood: Right, and by "your time" you mean "my time".

Spolsky: [laughs]

Atwood: You won't be spending any time doing that, it will be all me. But that's okay. What do I do? I just sit here and blog, right? It's something for me to do, that's good.

Spolsky: Well.. yeah.

Atwood: It's also fun, lets be clear. We talked about puzzles on the previous call, like how I hate to have.. like "How Would You Move Mount Fuji?", I think that's a total load (no pun intended).

Spolsky: Yeah

Atwood: But this kind of problem, I think is awesome. 'Cause I love nothing more than to spend hours working out how we should deploy our server structure, you know? Figuring out this deadlock issue was even kinda fun. Even that cookie hack issue I talked about, that was kinda fun to figure out.

Spolsky: Yeah

Atwood: 'Cause we had to go through the logs. We had to figure out where they were coming from. Where they got the script in the system. This was fun, this was like detective work. I mean, don't get me wrong.

Spolsky: Mmm hmm

Atwood: It's still annoying, they should have told us. And I also, let me give you an example. I learned about, and I think I knew this I just hadn't flipped the bit in my mind to make it significant. There's this thing called the HTTP only cookie, which we weren't doing which was a huge huge mistake. I'm about to write a big blog entry on this. You should pretty much mark all your cookies as "HTTP only". And it's enforced by modern browsers. So if somebody is running on an old browser, yeah, you're hosed. But if your running on a modern browser like Firefox 3 or IE.. even IE 6 actually does this. This is one of those Microsoft innovations, believe it or not. You can tag the cookie with "HTTP only", Javascript cannot get to it. It's enforced by the browser.

Spolsky: Ahh

Atwood: Oh yeah, it's huge. It would have totally protected us in this case, if we had the foresight to have this set.

Spolsky: No but, I mean wait.  If Javascript can't get to it... Oh, I see, I see.

Atwood: Yeah, because it's the script that's actually doing all the work, that's running in their browser. It's not like, you know, you're there with a sniffer, you know? It's just a Javascript. It's totally lame. What's so lame about, it's like it's such a lame attack, yet it's so effective. It's just scary.

Spolsky: Yeah. Because everything depends on cookies. If you have any kind of persistent login, it's just a cookie.

Atwood: Yeah, it's.. oh yeah. There can never be enough publicity around this. Believe me, I'll be writing the blog entry. So before we get too much deeper on StackOverflow, I feel like we didn't get to any questions last time. Do we have any?

Spolsky: We have bunch of good questions that have been sorta sitting here in my inbox for several weeks.

Atwood: Oh uhh... Let's get to it.

Spolsky: Let's go to Ryan Cox.

[39:51]

Ryan Cox: Hi Joel and Jeff. This is Ryan from Raleigh, North Carolina. I was wondering if you might talk about backups and disaster recovery from the context of Fog Creek as well as StackOverflow. Thanks and keep up the good work.

Spolsky: I can start with Fog Creek. What we did. And it's really ...

Atwood: Let me start with mine. Mine is easier: "We are supposed to doing backups?"

Spolsky: Ya. So you are saying you are not doing anything right now?

Atwood: We're are actually doing backup of the database every night, but it's not transmitted anywhere. So go ahead.

Spolsky: It's on the same hard drive with all the other .. . You are not even transmitting it anywhere?

Atwood: Of course, why not. We are like Windows users. We don't know any better.

Spolsky: Oh, OK. So, you know it's interesting. While I mention this, because I am a little bit, in Fog Creek ways, way more conservative than that. And I just sort of, you know, for years and years and years, I mean literally the first year we were selling FogBugz, Michael said: "Why don't we just make us available on the web, just like an on-demand-service." And it went through, these things went through three different names, before we find it. First they are called ASPs. And then they were called SAAS's.  And now they're called on-demands.  And when the third name came around for the same thing, then we finally did it. But the reason that I kept resisting doing it, is that we were a small bootstrap company and at first we literally could not afford a colo facility. We had a T1 to the office. And actually our public facing web server was an old Thinkpad laptop with a broken screen. So, for servers you don't need a screen and I kept that on all the time and it would get kind of hot. So I set it up on a little, like a plastic dish rack, like the thing you put your forks in. What are the things called? A fork holding thing. Just to get some ventilation around this old laptop. And that was our server. And I just could not ... I don't know if many people have done it, but I just did not have, could not with a clear conscience, sell people that as a hosted on-demand service for which I take money, because I knew that if anything happened and I had no recourse, you know, if the T1 went down or the server went down, or whatever, I would feel terrible. And I wouldn't be to deal with all the people calling up saying "where is my data?". So, we didn't actually go live until - and I became kind of more and more scared of this the time went on. And so we didn't finally go live with FogBugz on-demand until we had an infrastructure with almost no single point of failure. It has single points of failure, but they're very rapidly going away. So, our current infrastructure looks like this what we talked earlier about: Two web front ends and a SQL backend. And that makes what you may want to call a cluster. And we have one of those in New York and one of those in Los Angeles. In separate data centers, actually on separate backbones. And, what we do is, we put - both of these are live at all times - and we put half of our customers on New York and half of our customers on Los Angeles. And you sort of get assigned at random, although we probably should be looking on where you are coming from and try to get you to the lower latency data center. But we just assign you at random and then the SQL-Server instance that we have running in each of these data centers is doing regular backups. And it's actually shipping those backups to the other site where they are being restored and made ready as warm standbys. Not quite hot standbys because they are live right away, but they're warm and with one command the become the live site. And the way you do that - and SQL-Server is pretty good about this - is: you do a full backup and then you ship that over (and that's large). And then you load that up and restore that backup in the other data center. And then, with some frequency - and I think we are doing that at least every several hours - you do what's called a transaction log, which is kind of an incremental backup of what has changed since the last full backup. Or since the last any kind of backup. And then you get those very small files and you ship those over the net and it doesn't take any time at all, because it's very small, and you restore those too on top of it. So, basically we got a couple of things going on. One is: we got regular backups happening locally and those backups are also being shipped to whole another datacenter where they're being restored and we are almost ready to switch. And I think if we had a catastrophic failure of an entire data center, the trouble is, it would take us at least a couple of hours to gather enough information to find out that that data center is not coming back. Because if it's gonna come back you don't want to move everybody on to the other data center. Because it can then be expensive to re-partition them later. But if that data center actually blows up, and we know that it's not coming back, probably within a few hours we are up and running on the other data center.

[45:01]

Atwood: Well cool, and that's critical for you guys, because you're doing hosted services.

Spolsky: And yeah, and it's also mission critical data in the sense that, to a software developer. You know, their project schedules, and their list of bugs and list of features that they want to work on. You know, you can do without that for an hour or two, but after that you start to become really ineffective.

Atwood: Right. Well unless you're like me and just keep it all in your head.

Spolsky: Yep.

Atwood: Very effective.

Spolsky: Somewhat.

Spolsky: And with StackOverflow on the other hand, it's kind an optional service, you know. If the service disappears for a day, well: You didn't have it last year, you don't have it this week. Sorry.

Atwood: Yeah. No, I think there's actually an important distinction point there which is that, once customers go in put in data there for creating their own little worlds in there, right. To a limited extent people are doing that on StackOverflow. But we're talking about like what you said, like lists of bugs, you're outsourcing part of your business to Fog Creek.

Spolsky: Uh huh.

Atwood: There's no part of your business that that's going to go onto StackOverflow. Nobody's really in the business of answering questions,  programming questions.  So, um I think, obviously it's much more critical in you guy's [Fog Creek's] case.

Spolsky: On the other hand..

Atwood: That was a great question.

Spolsky: You would really hate it if the entire StackOverflow database just disappeared.

Atwood: Well yeah, it would be very dreadful.

Spolsky: You'd be struggling to recreate it from the Google cache.

Atwood: Yes

[46:20]

Spolsky: So what I would do in the case of StackOverflow actually if I was you is find some storage somewhere. It can be Amazon, it can be... Well heck, your server is not in your house, right?

Atwood: Well actually, we have, I have access to another hosted server at this vendor. So I can get to it by IP address. I can just copy it to the other server.

Spolsky: No, but if it's the same vendor, it's in the same room. So that doesn't...

Atwood: I guess that's true. You're right.

Spolsky: What I would do is set up SQL Server to do a full backup, a daily full backup, and then like hourly incremental backups. And take those backups and ship them to a somewhere in a different city. Just the files.

Atwood: Right.

Spolsky: That's not real hard work.

Atwood: Yeah, the Amazon EC is actually pretty cost effective in terms of the storage cost per byte over time.

Spolsky: That's one way to do it. Or you could even get a simple shared hosting account or one of those big drive things. Or heck, just email it to your Gmail account.
[Both laugh]

Atwood: Now you're talking like a Windows developer. Now you're using words I can understand: "Just email it your self. How about I just fax it to myself?"

Spolsky: "Email it to your Gmail account." Ahh...

[47:35]

Spolsky: Anyway, there's a lot of places to store just bytes. And that would be a good thing. And a very very important thing to remember whenever you're thinking about backups, and I always forget this. I always forget this when I try to design a backup strategy. There's a lot of things that can go wrong, but there are two that are very very different and you have to protect against both of them.
Number one is that your data disappears, and you need to get back to the data you had before it disappeared. And in that case, what you really want is something as close as possible to a mirror of the data in a different place. And so, a hard drive fails: Oh good, I've got a mirror in another city. You know, terrorists drove an airplane into my data center. Okay, I've got another one at home right here. And you want that to be as close a mirror as possible. And for that type of backup, obviously you want it to be as fast as possible. Like literally, you couldn't: A backup every minute would be not even enough because you've lost a minute worth of data when thinks go down. So that's the mirroring.
But, the other type of thing is that you've screwed something up. You went into the database and you typed a command, deleting a very important table. And you need to be able to get to how the data was in the past. And if you have a really good mirroring system, the mirror also has the table deleted. And you're screwed.

Atwood: Right.

Spolsky: So, you need to take into account both. How do I get back to something that I screwed up, when I screwed something up? You know, I have most of my data but I did something wrong to my data set.

Atwood: Uh huh.

Spolsky: Combined with, what do I do if my hard drive fails, or my server disappears, or blows up, or burns down, or something like that. So you have take both of those into consideration. And then there's all kind of, if you really want to get paranoid, you can go deep into: What happens if I notice at some point that six weeks ago I made a fairly crucial mistake and deleted some stuff, and I no longer have a backup that's old enough to go get that back. You know, you sort of have to... There's all kind of much less common problems. Or, what if you discover suddenly that a hacker got into your system nine weeks ago and has left, who knows what they're corrupted. You can't roll back to the state of the system nine weeks ago, because you'll lose nine weeks of questions and answers. That would be pretty bad.

Atwood: Right, like somebody actually went in and deleted all of your votes. Which actually happened to us. [laughs]

Spolsky: It's not always possible to solve those cases obviously.

[50:00]

Atwood: Right
So one of the reasons I actually started to doing the daily backup in addition to being just obvious

Spolsky: Mh hm

Atwood: was actually well, I've I've been working a lot on StackOverflow like I dream about it like I'll..

Spolsky:  I know

Atwood: It's pretty much consuming my waking life and my sleeping life.

Spolsky: hehe

Atwood: Uhh, so I have this really strange dreams.  And I dreamt that somebody had injected and like drop one of our tables.

And I woke up and I was just frantic and I like run over to the computer and I like went to the backup plan generator and...

Spolsky: yep

Atwood: ...immediately ran a backup plan. This is like a few days into the beta. So it was like it was just funny how, yeah.

Spolsky: Ah hm

Atwood: So yeah backups are good.

Spolsky: Backups are good.

Atwood: What's another ah hm.
Lets do another

Spolsky: Lets do another question.okay

Atwood: yes

Spolsky: This is a good one this came up on StackOverflow as well.

...

[50:45]

Ryan: Hi Jeff and Joel.  My name is Ryan. I’m a software developer from San Diego, CA.  In developing a database-centric application, designed for hosting multiple clients, I’ve always felt that it was better to use a single database with proper indexing to store the data for every client instead of using a separate database for each client. There are, of course, obvious exceptions to this, such as when you need to give clients direct access to their own database.

Now Joel, I’ve heard you mention that FogBugz uses one database per client instead of a single database for all clients. Why did you choose this structure?  And how do you go about providing updates and enhancements to all databases efficiently?  Thanks.

[51:22]

Spolsky: Yeah… So this is easy and I’m sure that the way we did it is right because what’s interesting is when you go to some of the other on-demand vendors who will remain unnamed who have on-demand products, this whole idea of having a multi-tenant db versus a single-tenant db is something, like, they’re all working very hard to get where we are, where every client has their own private database and they’re trying to basically… and that’s taking a lot of work.  We just started out with that architecture to begin with.

And the number of reasons is huge. First of all, let’s say that you have a database like in the case of FogBugz you’ve got a whole bunch of tables in there that are kind of top-level tables.  There’s project, bug, bug event—bug event’s not a top level table—but they’re top-level in the sense that they’re not somebody’s foreign key somewhere. And each of these tables would have to have that client ID column in it. So the size of your database would immediately grow dramatically because you would have to have the client ID everywhere, in every single, or you’d have to get it out of some kind of join somewhere to find the client ID.

And then once you did that, the chances that you could accidentally mistakenly write some SQL that showed one user data that belonged to another user just because you forgot to check the client ID or you forgot to filter appropriately in every single case, that you just constructed some SQL query wrong, is a very real risk. And if every client has their own database there’s zero risk of that happening.

So the clients are much, much happier when they know that their database is completely isolated and that nothing in their database is in any way intermingled with any other customer’s data.

[53:09]

The next is backup.  Clients… when you have an on-demand service… clients will often ask to be able to download backups of their data.  It’s their data and they want to get to it, and they want to make sure that they’ve saved it with whatever precautions they think are worth doing, in case you go out of business or strand them in some other way. Or they may just migrate off of your service and they want to be able to be given their data.
So for us to give a customer their data, it’s just a matter of issuing a backup command to SQL Server, taking that backup file and giving it to them. Or… actually I’m not sure if we’re doing that. I mean, the other way to do it is you just detach the whole database, copy the physical file and then reattach it.  And then you have a SQL file you can give them.  And you don’t have to go through and figure out which rows belong to them and which rows belong to other people and correctly filter those out. And run any kind of risk like that.
So it’s just monumentally cleaner to do it per client. And I’m not really sure why people think you should have everything co-mingled in one big database, what they call a multi-tenant DB. And I think that probably the reason they think that is that sounds like good data normalization techniques somehow, or that sounds like the kind of thing you do with data normalization. 

And it really depends on the scenario.  If you’re giving customers their own instance of an application, I think it makes much more sense for them to have their own database.  If you’re giving customers their own account on a larger system where the data needs to interrelate somehow or connect, then maybe it makes more sense to have it combined.
So for example: with FogBugz we’re giving people an entire application and it’s their data and it’s totally private. If you had an application that was—I don’t know—maybe it was like an internet calendar, that lots of people on the internet could use to make events and stuff like that and coordinate events with each other, then in that kind of case it might make a lot more sense to have it be a single-tenant…

[55:07]

Do you want to add anything Jeff?

Atwood: No, I think that’s totally a question for you.  I’ve never done it, so you’re the expert.

Spolsky: It's a actually kind of interesting that there are whole classes of things that people don't do anymore. It is actually kind of rare to run a software application for people. What people are doing is they are building websites these days and if they are building an application they are actually building it for their corporation internally and, you know, its like I'm going to do the payroll application and its going to be internal or I'm going to do the vacation tracker application and it is internal and there is just one of them and that's fine and when they are making websites for a lot of people the data that individual customers may have may be relatively shallow.

So, for example, take, for example, uuhhhh... What is an example of shallow data? Tripit. Tripit.com is a website where you forward them emails of the itinerary of your trip like those confirmation e-mails you get from airlines, hotels, and car rentals and stuff. You forward those to Tripit and they see your e-mail address and they build you a nice little web page that has your whole itinerary with everything all neatly sorted out, with little maps of the cities you're going to and its quite cute.

Check it out at Tripit.com and in the case of Tripit they have shallow data because for every customer they have a very small amount of data and they have an awful lot of customers and so it wouldn't be practical for them to create a database for every single customer and it isn't necessary, the need for privacy and security is not there and customers are not likely to call up Tripit and say, I would like to download my database please.

You know, there are just three itineraries in there, what exactly are you downloading? So with something like Tripit the need for customers data to actual intermingle is higher. For example, on Tripit I can go and say, "Hey, show my Dad this itinerary" and then he gets a message and can look at his itinerary next to my itinerary and compare itineraries and that is the sort of thing you need to have everybody in one database to do elegantly.

So, uh, there are different use case scenarios but FogBugz is actually an application we are hosting for a bunch of different people and giving everybody their own database apparently makes everybody the happiest.

Alright, I'm going to move on to one more question here. Here is Phil Howey.

Howey: Hi, Joel and Jeff, it's Phil here from Auckland, New Zealand. Got a question for ya. Do you ever suffer from having to maintain legacy projects with old code that no one wants to update and if so how do you balance that with keeping your programmers interested and using the latest technology? Thanks

[57:51]

Spolsky: [laughter]

Atwood: I think that's another question for you actually. [laughter]  'Cause we're using all the new stuff -

Spolsky: You are, because you're using the the new MVC, which isn't even shipping.

Atwood: Yeah, that's true, and then the LINQ -

Spolsky: You've got LINQ

Atwood: .NET 3.5

Spolsky: Very cutting edge

[58:08]

Atwood: Yeah, it's fairly sexy, at least from a code perspective... That won't last.

Spolsky: My whole career, except for this, has been about legacy - crap... and technology. Ahh... And we do sort of struggle with that. Like FogBugz was written in classic ASP, which was VB Script, and it became a very large code base, and there was never any real good motivation to change the language it was written on. And we got to the point where we had to have our own compiler, and I've talked about that in the past. So... Honestly, people like working on compilers.

Atwood: Have you talked about that in the past, I don't know if I've ever heard anything about that... I don't know if I can even keep a straight face while saying that.

Spolsky: [laughter] I won't go into that. Ahh, there are a couple of things - one, I don't think legacy code is necessarily boring, sometimes I think it's a fun little puzzle, especially if I don't have to spend a whole lot of time in there. And sometimes it's more fun to just dive into some like horrible, horrible code base, and try to figure out if you can breath some life into it, or make some small change without getting burned. Umm... that may be kind of a fun project, other than that if you had -

Atwood: Wait a minute, let me interrupt you just one second here. So I think there's two classes of really horrible code. There's horrible code by good programmers, and there's horrible code by horrible programmers. And these two things are very, very different.

Spolsky: So when you say horrible code by good programmers, you mean good programmers working in some like ancient VB Script type.

Atwood: Yeah, exactly. Like there's just a huge range. The reason I say this is because we have some horrible legacy code ... I'm kind of doubting, at FogCreek, with the kind of programmers you hire, you're ever reallying going to have the kind of horrible code to the point that.

Spolsky: No, it's very clean, it's very clean.

[59:57]

Atwood: It's conceivable that I could go into your code, say okay, it's in Wasabi or some crazy thing I don't understand. But like wow! They actually thought about what they were doing. Because when you look at a lot of the old code that's really bad, it's like programmers who just weren't thinking about what they were doing, like at all. Like you can't even. Like what was going through their minds?

Have you seen that famous picture, well famous now, on the internet, where it's like two pictures of a door for code review? Good CODE REVIEW - WTF, WTF, WTF. Bad CODE REVIEW - it's like 50,000 WTF's. Right? Even the good code is bad, is my point. And there's these huge degrees of bad. Like so, I think it's conceivable that what you're saying is true, at FogCreek you could have legacy code that's reasonable to work with. But I think you have to realize that.

Spolsky: So it's sort of like... The first question is, is the code bad, or good? Secondly, is the technology legacy, or not legacy? Is it an old technology that's just not interesting to work on, or is it restrictive, or very painful? And then there's also like, is this code that lives? Is this your business? Like I think a lot of developers would not want to go work for a company that pretty much sold a product that was written in VB 6, that had no plans to port, and that's all you're going to be doing all day, working on VB 6 code for the rest of your life because that's what their main product is written in, and they're never going to port it. That is hard to get people interested in doing. Now there's a difference between that and "We have this old VB6 app; it's working fine; nobody's ever opened it up, but now we need to change something about the sales tax rules because some new law was passed because some law was passed. Can you go in there and make that one tweak?" And so that can be kind of fun, but if you're really bogged down in some kind of old technology. We aren't, by the way, we just wound up taking over the compiler so that the compiler could be a compiler that we wanted to work with rather than a compiler that we were stuck with. But not everything's like that, and I know certainly a lot of places. A lot of the top investment banks in New York are kind of famous for hiring programmers, and they've got a lot of legacy code and a lot of boring code, and it's not a great working environment in terms of how well programmers are treated, and there's all kind of things that are wrong with wanting to be there as a programmer. But on the other hand, these investment bankers have a lot of money, and they want really really good programmers who went to Harvard. So, what they end up doing is giving their programmers relatively free-reign to mess around with new technologies sometimes, and it's not uncommon to see thees big investment banks. You talk to a programmer at (this has happened to me so many times now), you're talking to a random programmer "Oh, where do you work?" "Lehman Brothers" (the names have been changed to protect the guilty) "What are you working on?" "Well I'm working on this team and we've got all this legacy Linux code and we finally decided to port it to the latest version of .NET and C# 3.5 and my boss wants to do something with F# and so we're on this major 2-year project to re-write everything on this whole new platform." And they're all very excited about that and I don't want to tell them that, that whole Linux thing was a major re-write from the previous version, which was all in, lets say, ASP.NET, or the original ASP; VBScript or something. Which in itself was a huge re-write of some system that was in Java; Which in itself was a huge re-write of a Windows app written in C++ with a user interface created using Visual Basic; which in itself was a re-write of some Excel macros; which itself was, well, that was when the business was invented. So, every two years some new boss comes along and they just decide to re-write the whole thing in some new language. And this costs a great deal of mony and it's just not worth it and it's a complete waste of time, but these investment banks are willing to make this investment in letting people screw around with the fun new technologies because it's the only way they can hire the programmers of the caliber that they want.

[1:04:05]

Atwood: so you're saying there's a balance there, it's like if you're going to ask programmers to do janitorial work give em something else, kinda like google, like the mythical google 20 percent of, OK you have to work on this really boring ad platform that we have and then the other 20 percent of the time you have...

Spolsky: (interrupting) yeah but nobodies gonna tell you that you can't have a big project to port everything to C#, and then the other thing you see at these investment banks all the time is that they'll constantly, in any one of these investment banks somewhere you can find all the elite programmers working on some kind of an infrastructure project that has nothing to do with finance, it's just pure software engineering. So it might be the 'Universal Login Architecture', it might be the 'Universal Data Transfer Architecture', it might be the 'Universal Risk Management Architecture', its some kind of thing that, that you can buy off the shelf, theres probably an API for it in Windows if you just look hard enough, or an app you can buy, but you're writing your own because that's whats interesting to the top programmers and they have somehow convinced the business people that thats how it has to get done - and er, and er, and it's fun. So they may have you know, the mediocre people get stuck working on some kind of legacy code but all the top people find some way to write a universal login system with umm, that never ends up shipping that they spend 2 years on written in umm, I don't know, Haskell or F# or IronPython or some, you know, something fun - and cool.

Atwood: oh come on, Ruby.

[1:05:30]

Spolsky: I haven't seen Ruby yet, but it's going to start showing up there in the investment banks. It's not their flavor.

Atwood: You bring up a good point. On previous podcasts people were asking, well, why do Jared and Jeff work for you, for slave wages on StackOverflow? And truly, it's like -

Spolsky: Slave wages.

Atwood: You couldn't even survive on what I pay them. And I think one of the reasons they want to do it is because they know me personally. There's a personal relationship there, hopefully a good one. And second, because this is a really cool thing we're building. I love building this thing, and they love it too. So this is their twenty percent project, they have full time jobs, right? They're at very flexible schedules. But this is their twenty percent project.

[1:06:24]

Spolsky: And you know what? Three years from now, they're gonna be famous. Not that that matters.

Atwood: Possibly, I mean it's... Even if they're not famous, it's like you said, this is the biggest system we've ever worked on, easily. Right? How many people is this system going to touch, vs. anything else, anyone of us on this project has ever worked on? So from that perspective it's exciting.

[1:06:42]

Spolsky: It's just really awesome to bump into people that use your products.  That was one of the best things about when I worked on Excel - yeah, I don't mean famous - it's like, you tell people you were on Excel in the early days - it wasn't even such early days - I worked on Excel 4 and Excel 5.  But they're like "Oh Excel! I use that every day, I love Excel!".

Atwood: Yeah, it's cool.  It's definitely cool.  And we've talked about this before, but it bears repeating - it's like - to me, as a programmer, the biggest complement you can get is for people to use your stuff, man.  I mean, that's the whole reason we do this. So if you can position yourself, it's like, why do people take a job at Microsoft?  It's like, there's a lot of reasons people take jobs at a lot of different places - but at Microsoft you could potentially work on something that a lot of people would get affected by.

Spolsky: That's right.

Atwood: Like if you wrote some stupid bit of code... What about the guy who wrote Notepad?  O.K., notepad sucks, right?  Nobody will argue this.  But I mean, how many people use Notepad ALL THE TIME?

Spolsky: Notepad is just one of those edit controls - and then, lately they had to add like 60,000 lines of character encoding logic to it - but that was very recent - to make it super-uni...

Atwood: Or like, what about the guy who wrote Minesweeper - with his crazy little cheat codes?

Spolsky: I wonder if that's Wes Cherry? (sp?) He wrote - no Wes Cherry wrote Solitaire.

Atwood: Yeah, Solitaire.  I think I saw an article on the Minesweeper guy, but you can be really famous for writing this stuff if you get in the right business.

Spolsky: You know what - I hate to say this - that's definitely a good reason to work for Microsoft, if it's the right team.  On the other hand, sometimes I'm trying to use some feature somewhere in some Microsoft product, and it's just not working, and I'm searching for it on the Internet, and I realize that nobody else in the world uses this feature - 'cause if they did, they would be talking about this - whatever this horrible bug is...  We're way overtime - so if any of you know how to publish Outlook 2007 calendars from public folders in a reasonable way, that's what I'm talking about...  This is obviously a feature that nobody has ever used, 'cause it doesn't even work - and there's nobody even complaining that it doesn't work on the Internet, that's the real problem - the real sign that nobody's using it.

[01:08:44]

Spolsky: Any announcements? Anything else you want to close with?

Atwood: No, just September 3rd is the planned date to launch

Spolsky: I think that honestly, I know that you've got things on your list. But I know we'll be ready. The only thing I don't know about is what's going to happen when the world hits us. I'll try to write something on Joel on Software Jeff, and you should too I hope on Coding Horror explaining at we think at least the site is kinda suppose to work. So that some of the hordes that hit it on the first day will have something that they could possibly read if they are at all interested. And find out a little bit, about how it's suppose to kinda maybe work.

Atwood: Right

Spolsky: And we don't have a bunch of newbies hitting the site, doing the wrong thing.

Atwood: Right.  So final notes.  I'll do the final tail-out.  So, if you want to submit a question for us to answer on the next podcast.  Do an audio recording, 90 seconds or less and mail it to podcast@stackoverflow.com.  Also, we have a wiki where we have a transcription of these podcasts.  And I'll make the same offer I always do.  If you're desperate to get into the StackOverflow beta, we have a huge waiting list actually.  If you do a little bit of transcription on the wiki, I will bump you and you'll get in early.  And you can get your beta badge for being a beta tester as long as you post one question and one answer on the system.  And beyond that, see you guys next week.

Spolsky: See you guys next week.

[01:10:09]

Outro, advertising