The StackOverflow Podcast

Intro, advertising

[0:59]

Spolsky:   Hey, what's up?

Atwood:   Just working on StackOverflow: the usual things.

Spolsky:   Really? [Laughs]

Atwood:   Yes. Contrary to popular opinion, we do actually do work on the project. I know you don't believe me, but it does actually...

Spolsky:   What is this project again? Is it a kind of a of podcast? [Laughs]

Atwood:   Yes, its a kind of a website we're building.

Spolsky:   That reminds me, we should probably take some time to explain to some of the new listeners, the IT Conversations listeners, who suddenly got our podcast shoveled into their iPods without asking for it a little bit about what we're building and what the point of the podcast is.

Atwood:   Oh sure. Absolutely. So StackOverflow is a joint venture between me and Joel where we are trying to combine our communities, our respective sort of... programmers and those that love programmers. And actually build a website that serves that community. Its sort of built by them, for them. And it's specifically to address the growing number of programmers that aren't really learning programming from reading books any more, they're learning programming by doing stuff - loading their IDE or TextMate or whatever they use - and just trying things.  And when something doesn't work, they then do a web search to find out "why didn't that work", "how can I learn more about this" [Spolsky: Yep.] and so StackOverflow is really there to feed those searches, and it's not by us, me, or you, but really everyone. So as little problems come up you'll post a question on StackOverflow and it's a very low friction, very simple website, not a lot of long-term commitments, no sign-up. Very Wikipedia like in some aspects, and our hope is that you'll find... be able to post your question and also find answers to questions you have while programming on the site.

Spolsky:   Cool.

[2:48]

Atwood: Yeah, and...

Spolsky: Yeah.

Atwood: I have my movie pitch, hold on, my movie pitch, which is: It's sort of like Wikipedia...

Spolsky: The elevator pitch? 

Atwood: [Laughs] It's like that, but I like to call it the movie pitch, you know, they always talk about movies it's like: Driving Miss Daisy meets The Predator, right?

Spolsky: Right.

Atwood: So I'm gonna call it...

Spolsky: "It's The Graduate ten years later"! [Laughs]

Atwood: That's right, so it's like, Wikipedia, meets blogs, meets Reddit/Digg. And also a bit of Experts Exchange but we have very ambivalent feelings towards Experts Exchange.

Spolsky: We don't want to mention them.

Atwood: Yes. Sorry. I've already done it.

Spolsky: First of all they put that annoying hyphen in there, so it used to be Expert Sex Change and then they put in hyphen in to disambiguate

Atwood: Yeah. They should just embrace that identity, I think.

Spolsky: Exactly. That was sort of kind of funny.

Atwood: [Laughs]

Spolsky: Yesterday I had an error message and I tried typing in and went to Experts Exchange and it said "The answer is only available if you pay" but of course, sometimes, if you just keep scrolling down, you'll see the answer anyway.

Atwood: Well, I will say that some of the people at Experts Exchange do have a sense of humor because they actually saw what we were doing, someone did in their community and invited me to their conference which is, like, in August and I forgot where that is, actually...

Spolsky: Where is it?

Atwood: Yeah, that's kind of important. [Laughs] But I thought I might go because it seams like at least a couple of people had, you know, some sort of a sense of humor and they realised "Hey, you know, we're not all necessarily competing, it's a very large pie", which is the way I still view it.

Spolsky: You know, a lot of the people that contributed there are gonna contribute to our site. Heh heh heh.

Atwood: Sure! Yeah, I mean it's... I think our site is gonna be better because it's going to be very low friction and very simple and clean and just, basic. So I think we'll distinguish ourselves that way, if by no other way.

Spolsky: We are not gonna be selling memberships.

Atwood: No. We won't be selling you anything. There will eventually be ads (just to be clear) because this is a... unlike Wikipedia this is actually a for profit venture but... you know, in a very tasteful way, it's the way I say it; and in a way that serves the community, and not fights it.

Spolsky: Right, in an earlier episode we had actually talked about trying to make sure that the questions and answers were in some way Creative Commons licensed or something.

Atwood: Oh yeah, uh, cc wiki is what we're going to use. And actually, somebody emailed me who had grave concerns about how we were going to license and I emailed him and said "hey, cc wiki is in the beta, it's in the footer of the beta, and what do you think of that?" and I guess he hadn't seen that particular one but he seemed very encouraged that we were thinking about it and doing something about it. And that's really the intent of these podcasts and the blog and all this other stuff, and why we talk about it before it's actually available is we wanna get feedback from the community on what it is we're supposed to be building as we build it. And we don't want that to go on too long obviously, we don't want to have like a Duke Nukem Forever type situation where, you know, it never actually ships. But, uh, I don't think we'll have that problem.

Spolsky: [Laughs] That's funny, I, you know I was pretty sure, that's an excellent opportunity for a rat hole, that Duke Nukem Forever, I was, I'd been convinced for the last decade or so that they were just pulling one over on us, pretending that there was another version of Duke Nukem coming out and not really doing anything, and every once in a while, just for kicks, they'll leak something somewhere to somebody.

Atwood: You know, in a way I'm going to be disappointed when it ships because it's become the benchmark for all software of like "well, at least we shipped before Duke Nukem Forever". Like, the Wine project, did you see this was news? The Wine project, which is like a Windows emulator under Unix ...

Spolsky: Yes, they declared that they're version 1 now.

Atwood: ... actually shipped version 1.

Spolsky: Yeah, yeah.

Atwood: That was 15 years.

Spolsky: Really?

Atwood: [Laughing] Yeah, 15 years. It's an old project. I remember a friend who worked at Sun, gosh this was in like '93, and she was telling me about the Wine project. And it wasn't a new project then.

Spolsky: There was another one that was made by a company called Bristol Technology, you remember that?  It was like a way to get your code to compile under Unix, if it was Windows code.

Atwood: I don't remember them.

Spolsky: And I don't know what happened to them, I think Microsoft sued them into oblivion, uh, because, I don't know, let's look up "Bristol" on the web.

Atwood: There's a wine that's called...

Spolsky: Bristol.

Atwood: It's probably not the best search term to use. Wow.

Spolsky: They're just gone.

Atwood: That's what happens when Microsoft doesn't like you man. They, they're just gone.

Spolsky: They were bought by uh, yeah, there was, they were acquired by HP, but it looks like they have long since given up their original product, which was kind of like a commercial version of Wine, essentially.

Atwood: Ah, I see.

Spolsky: Umm, ok, let's see I have some, did we have something to talk about initially? Oh, uh, just to conclude then what this podcast is, for those of you who are new to the podcast, is, since Jeff and I are on opposite coasts, Jeff is on the left coast and I am on the correct coast, uh, we uh, meet virtually every week by having about an hour long phone conversation just to check up on the status and talk about any design questions that may have come up or, uh, things that are going on with StackOverflow and just generally just to chat, you know, and we thought it would be fun as long as we're having these weekly phone conversations to start recording them. Which is what we did. In the first episodes you can actually hear they sound like phone calls, we've since switched to Skype which has higher audio quality. And, uh...

Atwood: And you're using a, you're using a new fancy mic. right?

Spolsky: I'm, yeah, I'm using one of these like fancy radio broadcaster mikes.

Atwood: Awesome.

Spolsky: I can't hear the difference. Can you?

Atwood: I can hear it. Yeah, I definitely can. 'Cause there was a little bit of breath-noise on the other one, but it sounds, it's better.

Spolsky: It's true, we've also eliminated the air conditioning noise that was, it's pretty loud in our office.

Atwood: Yes. So, I also have, I want to make a confession. The way this works is we record 'em and then I listen to them and make semi-detailed notes about sort of what we talked about, and some, I like to hyperlink a bunch of stuff so that people who don't want to listen or just want to see the topics can just follow along. Plus, I find it useful as well to go through the notes...

Spolsky: Yeah, I think it's incredibly useful, those show-notes.

Atwood: Yeah, so, I want to make a confession, so as I'm listening to the show I realize that, and I've gotten better with this and I think there's other reasons this happens, I realize I'm not hearing everything you say. [Laughs]

Spolsky: [Laughs]

Atwood: This is probably not a revolution, a revelation, to people who, listening is really hard. And in my defense, a lot of times like I don't want there to be dead air as we're talking so I'm trying to queue up things I want to say so that there's not some long pause while I think about "oh, what should I say next".  So that's my defense.

Spolsky: Jeff, you're not listening to me?

Atwood: Well, it's strange I would hear you say something wow I, you know after the fact I might go "wow, I totally did not hear Joel say that..."

Spolsky: Just, I think I have to have a test. There's gonna be a test at the end of this episode [laughing] on everything I said.

Atwood: It's really hard, that's really the lesson, is like, "do you really listen to people?"  It was really kind of shocking to me when I did that, and I've really tried to do better because I was shocked I was like "how can I not be listening to someone who's talking to me?" But I think, you'd be surprised, I mean just recording yourself and listening to it this way is, is a little bit shocking in what goes on.  And, I have improved, I've noticed it happens less often where you'll say something and go "wow, I wish I had heard that the first time 'cause I would have responded to it."  Um, so that's my confession of this particular podcast.  We have a lot of progress on the project, so there's a number of things I could talk about there.  How many questions do we have queued up?

Spolsky: OK, uh, let's see, um, wait, you were saying something but I wasn't listening. [Laughing] How many questions do we have queued up?

Atwood: That's right.

Spolsky: I got, I got three.

Atwood: OK, we can probably...

Spolsky: Um, and also, I've also queued up, this [rimshot sound]. So if you want to tell any jokes, I've got a rimshot all set up.

Atwood: Ugh, ok, please, that's, we're only going to do that once ever right? That was the one time, and...

Spolsky: [rimshot]

Atwood: Ugh, yeah, I guess I asked for that, so.

Spolsky: There's a website you can go to.

Atwood: Yeah, that's great. Great. So now it's like a morning show.

Spolsky: It's called instant, InstantRimshot.com it's just got a big, gigantic red button right in the middle, and when you click it [rimshot]  Isn't that awesome?  If you ever need that, really quickly.

Atwood: That's great, we can't do that enough, I think, we should have a whole show where that's all we do.

Spolsky: Alright I'm done.

Atwood: There's a site like that, those are called single serving websites, website that you go to have very, very, that's one thing they do and it's ridiculously simple.  There's a site like that but it does that "wha wha wha whaaaa" sound (laughing) which I find much funnier actually.  My wife, uh, told me she used to work with someone and they talked about the transitions sound between this person's slide should be that "wha wha wha" because it's, their presentations were so bad, that would have been wholly appropriate for that person. 

So let's, let's talk about StackOverflow and what's going on.  So, first of all we did have a beta site up and very clever hacker listeners figured out the URL for our site - not that it's very difficult to figure out [laughing].  And we're actually interacting with the site, now, normally I would actually welcome this and I, I - don't get me wrong I think it's very cool that you guys are interested and engaged and want to help us - but, uh, the site is really not fit for human consumption at the moment in that we're rebuilding the database like pretty much daily.

Spolsky: How come I can't figure it out?

Atwood: Uh, I dunno, are you, maybe, you know, this is like one of those tests you talk about in your book.  Maybe you're failing your own, you wouldn't hire yourself at this point.

Spolsky: What is, it's not beta.stackoverflow.com?

Atwood: It is, that was what it was, but I moved it because...

Spolsky: Because people were figuring it out.

Atwood: And this is really for your own protection, I mean I, I want to wait until the public beta, or, excuse me the private beta.

Spolsky: OK, ok, ok, what was your blog post about this week on Coding Horror, not that you had only one. Specifically, June 14th, 2008. And the headline of which was "don't go dark."

Atwood: Well, I thought that was, yeah, you commented there, and I was like "that was a little unfair", have we really been dark? We have this podcast, we have a blog where we talk about what we're doing.

Spolsky: That's true.

Atwood: Uh, we're not really dark. Dark is like dark. Dark is like just a void where there's no idea what's going on, and I don't think that's fair to characterize it that way.

Spolsky: No, you're right.

Atwood: Um, but anyway, I do appreciate the interest, and actually anyone who found the beta site legitimately, email me, 'cause I will give you a special badge - a hacker badge - that will be on your profile on the real site once we have the private betas which I thought you might find fun.

Spolsky: We don't need no stinkin' badges.

Atwood: I think the badges are fun, I think people will see how they work and the, they're completely optional, you can just ignore them if you don't like them.

Spolsky: Badges?

Atwood: I know the movie you're referring to that's the Sierra Madre, Treasure of the Sierra Madre.

Spolsky: And it's one of those things that they never really say that the way you, you know they never actually say that.

Atwood: Right.

Together: It's a mis-remembered quote.

Spolsky: Let's see the official quote is badges, we don't need no stinkin' badges, it's "I don't have to show you any stinkin' badges."

Atwood: Yes. So, the badges, the way we're viewing them are complementary to your reputation score. And all this is again optional, you can ignore it, but it's for a way that for people participating in the system. The only importance it will have for these people is that you will not be able to treat the site like a wiki and actually edit everything until you get enough reputation in the system. So, we're like Wikipedia in that you can edit everything, but we're unlike Wikipedia in that you have to earn that ability a little bit by participating in the system normally for a while, and then after the system trusts you it'll say "ok, you can do this, 'cause you're, you know, you've participated." Yeah, I think it's a good balance, because we'll still have anonymous participation, to be clear, anonymous people will be able to ask and answer questions no problem, they just can't edit, 'cause we view edit as kind of a privileged operation.

Spolsky: I think it is gonna be terrific.

Atwood: Yeah, I think it's a nice compromise, I mean, I think it's a nice, sweet middle ground that will work well. So, we've done a lot of work around build, well technically Jarrod, one of my co-workers on this project, did a tremendous job on getting our build up and running.  We actually have a build script now that's checked into source control, so we can do, we can pass the Joel test, we could do a one step build that deploys the database, the code, and, actually even does unit tests, Jarrod went way above and beyond.  One of the advantages of the MVC framework, it's really easy to write unit tests for this site, because everything is, is a URL, like every action is a URL.  There's not a lot of post-back stuff going on.  So we actually have, as part of the build, it deploys the database, deploys the code, and then it actually creates users, you know (and that's just, again, URLs), and then it actually creates questions.

Spolsky: Don't you have to be a little careful to make sure that like those URLs aren't things that search engines and Google spiders and stuff will accidentally hit?  Like if you've got a button that's like "click here to delete all posts in this area" and then a spider comes along and clicks it while it's downloading copies of everything 'cause it thinks it's just a link.

Atwood: Yeah, I think that's not usually a problem, I mean I think as much as the web is just a random motley assortment of code I think that would be happening a lot.

Spolsky: It's not a problem if it's behind a form or something.  But, like, sometimes you have a temptation to, I think that's why there's this general rule that a hyperlink should never take an action.  Uh, only form submits should take actions.  Because, uh, you know the spiders will never try to fill in a form.

Atwood: Well, it's not that we don't have some post-backs, I think there are some post-backs, but they are greatly reduced relative to the typical ASP.NET programming model, the, you know, "pretend we're on a windows form" model.

Spolsky: And it's a GUI in VB.

Atwood: Yeah, yeah, so I think it's not entirely that way. But just enough so that development, I think, becomes simpler.  And one manifestation of that is these unit tests. And it was cool actually, last night I added a field to the database and I couldn't get it to deploy and I wasn't really thinking about this happening and Jeff - another developer - looked at it, 'cause I wanted to have another pair of eyes on it.  And he's like "oh, you're failing this unit test" and that actually blocked the rest of the deployment, which made total sense.

Spolsky: Nice.

Atwood: Yeah I know, it was very cool, so I want to give Jarrod credit for doing that. 'Cause first I was like "why are we wasting our time on", I am not, to be clear, I'm not, I love tests, but I'm not a test first guy, I just don't get that religion.  Um, I view it as, you want to have some of the site running, then sort of add your tests in after you have sort of a minimum level of basic functionality.

Spolsky: Well, of course, the temptation, I was talking to, who was I talking to about this yesterday - you?  Anyway, some

Atwood: It wasn't me.

Spolsky: [Laughs] there's a real tendency, you know, the one thing that all developers in the world have in common is that they will tell you that the codebase they're working on sucks and it's a big mess, and if only they were working some place else that was more professional the code would be nice and whatever.  And there's a real tendency whenever you're doing any kind of project, no matter how large how small, how much code, whether it's like 15 minutes of code that you're going to write right now or a day of code or a week of code, to do it in two phases.  And in the first phase you say "let me just get it working", because, if I don't have something working then there's no point if it's good or not, I'm going to do the minimum, you really want to get that reward of getting something up and running and seeing it working and then you can come back and make it fast, and you can improve the algorithm, and you can polish things, and you can restructure the code so it's clean.  And there is just a real tendency, and that's when you start to use cut and paste code, and all kinds of the anti-patterns from that great book called AntiPatterns, I dunno if that's on my list, but all of the things that you do to just get it working that you think you'll clean up later.  And then the joke is you never come back and clean them up later, and that's not because you're a failure as a human being (although you might be) but that's not why you're not doing those other things.  You're not doing those other things because once you have it working they suddenly seem a lot less interesting to you, you're like, you know, I've got a few pages of code, I don't even have to look at it anymore, 'cause the code now works, like, why should I even worry about that. In fact, it occurred to me, and I started to notice this phenomenon with code that I wrote on Thursday. I try to make like at least a day every week or two in which I literally do not open my email or web browser and just try and take a day and write code, otherwise I would never get any code written. And what I was writing actually is a little utility for the internal sales people here at Fog Creek to use to generate coupons. Uh, which is 99% of the coupons are somebody asking for educational discount on our software so we give them a coupon. So, um, we used to have this horrible way of doing it and I made a good way of doing it. And basically, somewhere in there the data structure works like this in the database, there's a row for each coupon, and then for every coupon it may cause various discounts to be applied to various products. So, like, I could give you a coupon that's worth 50% off on FogBugz for Windows, 50% off on FogBugz for Macintosh, 50% off on FogBugz for Unix, 30% off on Co-Pilot, or whatever the case may be. So, that's obviously a one-to-many relation, and so I've got to insert one record in the coupon table and then I've got to get the ID of that record and then insert a whole bunch of records in the coupon line, line of coupon table, all using that ID. And I was actually doing this from code, and as I was doing this it suddenly occurred, I was doing this by having two separate blocks of C# code, the code that inserts the coupon and then the code that gets the newly inserted ID and creates all those other rows. And I suddenly realized I could do this all from a single SQL statement, or maybe not one statement, the statements could be merged into one call. And that would make what is now about a page of code become about half a page of code, and it'd just be cleaner and nicer. Did I do it? No. 'Cause I got the thing workin', and then I went home. And there's just no reason ever to open up that code again. But if anybody does open it up, they're going to say "ugh, why didn't he..."

Atwood: Well, you don't have anybody reviewing your code do you? You don't have a multiple set of eyes.

Spolsky: That helps? [Laughs] Nobody, nobody.

Atwood: It helps tremendously, that's why I was really glad to add Geoff to the team.  Because having, working with another developer is really big. Just, you know, the value of having four eyes versus two is like, more than double.

Spolsky: Oh yeah, this isn't, to be fair, this is not really quote "production" code, this isn't code that we're trying to sell to somebody, it doesn't have to last very long, it's just a tool for our internal salespeople to use.  Once it works, and this is one of the things that I think people find depressing when they work in internal IT instead of working at a product company as software developers.  It can be real depressing because once you get the code working, the business case for spending any more time on it evaporates.  It's like, "we can now make coupons, so why are you concerned about whether there's, you know, one SQL call or two SQL calls there, don't even obsess about it."  And, uh

Atwood: Right, well, time to refactor, because, see, that is true, I mean, you know, you move on to the next project you're done.  But is code really done?  When is code done?  I mean

Spolsky: You could sit there and polish that little piece of code 'till the end of eternity, you could make the world's best.  But it would not improve your sales people's ability to make coupons, and therefore there's no business case for it and it's a waste of time.  On the other hand, if you make a company that's like the, the coupon company, and that's your whole business, is making it really easy to make coupons (I don't even know what that means), but, let's say this code was a part of our product then every time I make it better I'm going to improve my customers' happiness and therefore sell more copies.  So, when you're writing production code that you're going to sell, or code for the marketplace, or even like working on a website that the whole world's going to see, like you guys are, like anything - StackOverflow, Google, whatever - every time you make that code better, you make your business more successful.  Whereas when it's that secondary support code that's being used for some internal payroll application it just has to get good enough and then there is no return on investment for any further work.  So I think the people in IT departments are just really depressed.  Or, eventually they get kinda frustrated because there's a real strong business argument for them just not to work very hard or make their code very nice.

Atwood: Absolutely.  I mean, that's one of the key pieces of advice I think a lot of people eventually arrive at as a software developer is if you really love this stuff you have to work somewhere where software is the product, it's the thing you're trying to deliver to the customer, because if you don't then you're invariably going to get frustrated with the model.  And I think that's natural.  I think you just have to realize what kind of developer you are, are you, you know, just, the guy, the kind of guy who does it 9 to 5, it's your job, you don't really want to take it home with you.  Or is it like, everything in your life, like it is for me (which is a little disturbing) then you definitely want to work for a software company.  And I think that's a key decision you have to make.  And I've certainly given people that advice before that identified as developers who really just love the process of coding, it kind of destroys you inside after a while if you can't give the code the love that you feel it needs.  So, I think it's a completely valid point, and it's been brought up, I know you've brought it up, and it remains totally valid.

Spolsky: My time in the industry was working at Viacom.  So I worked on websites for MTV, the Sundance channel, Blockbuster, you know Viacom is just this big holding company that owns all these other media companies.  MTV is the most famous one.  And the code that I was writing there, I just remember like getting something that they wanted, somebody wanted like an automated FTP, and I wrote this totally overkill elegant system for synchronizing two directories using FTP from a laptop to a server and then launching some kind of a content management system in the background.  And it was just like totally overkill and they appreciated that, but then I documented it and I made a setup application and I made an administrative tool and I made six, there were six different manuals, depending on what your role was in using this thing, you had a custom written manual that described exactly how you worked with this tool.  And, nobody ever once looked at those, and the only reason I even did that is 'cause I was basically under employed there and just had nothing to do most of the day.  So I sat around polishing these things.  And, it was sort of frustrating to me that absolutely nobody ever looked at it, or used it, or [laughs] ran the code.  But then they certainly didn't appreciate the overkill that went into creating the promotional website, the internal promotional website for the tool, or the documentation, or the setup apps, or anything like that.

[25:56]

Atwood: Right. And that's the interesting thing, I think, about your history, because I always think of Joel as the Joel-On-Software Joel, but if you look at your history, it's really interesting: you followed in the footsteps, I think, a lot of typical developers who come up through internal IT. You had jobs like that. It's so hard for me to imagine you [Laughing] in that role.

Spolsky: You know, I used to have to wash dishes.

Atwood: Just doing, just, internal line-of business apps. Yeah, pretty much, you were like peeling, the programming equivalent of peeling potatoes.

Spolsky: But I actually did peel potatoes. I didn't peel that many potatoes. Onions, when you peel onions, ugh. I worked in the, when I was in the Israeli army I just spent months working in the, I don't want to say dining room, because it was a tent, with mud floors, in the desert. And my job was setting the table, and washing the dishes. Clearing the table, setting the table, washing the dishes. And that was like 18 hours a day: Setting tables, clearing tables, washing dishes.

Atwood: You know, it's interesting, a lot of your military - I've been meaning to bring this up and I guess we might as well since we're on it - your military background has been really fertile ground for you for writing.  And it was always interesting to me 'cause one of your favorite posts that you wrote was about learning how to clean the toilet.  When the starched super clean sergeant, or whatever rank, sergeant, said, "this is how you clean a toilet."  And then he proceeded to get down on his knees and clean the toilet. And you were like, "wow, and I was like that's such a powerful -

Spolsky: It really was, no, but I fell in love with that guy,

Atwood: - [...] Leadership"

Spolsky: Oh my God what a great guy. I was just like

Atwood: Yeah, I mean, no, that's leadership.

Spolsky: I would have gone to the end of the Earth for him.

Atwood: I've actually saw another anecdote that was similar where somebody had spilled something on the floor and the CEO of the company comes over and starts cleaning it up. Right.  Because the lesson is everybody takes responsibility for everything that happens. Just 'cause you're the CEO doesn't mean you get a pass on doing the things that need to be done.  And that's, I love that story, it almost makes me want to

Spolsky: It's not just that, it's also, like, for that CEO, it's like I am not above you, I'm not some special person that needs to be spoon fed that can't clean up his own spills. And also, you guys, we really do draw our org-charts upside down at Fog Creek.  And we see "management" as just being here to support people and in fact I was just talking to somebody who was recently promoted to be the team lead of FogBugz, who was talking to me today, and he's like, you know, he mentioned something which he thought was kind of frustrating which was that a lot of times when there are like little emergencies that just need to be fixed like there's a little bug here in some old version of some code, you've got to check out that old version, fix it, get it deployed to these four customers, whatever. These just like little emergencies erupt all the time.  And he winds up doing all that stuff.  And the reason is because at Fog Creek we totally recognize that we want our developers to be able to concentrate on one thing and just come in and concentrate on that thing and never get interrupted, and not have to multi-task.  And the only way to make that happen in the real world when there's all kinds of little emergencies cropping up all the time is that the emergencies get handled by the managers, so that the people can do their job, no matter how grubby the emergencies are.  And if you look at what me and Michael do here it's just a lot of like ad hoc, like, literally, right now I'm making decisions about the benches that will go in the coffee bar for our new office.  You know, stuff that you would think like, "hey why don't you delegate that stuff, and think about the big picture," but the whole reason I'm not delegating that stuff, is so that the staff here can concentrate on just doing one thing, whatever that thing may be.  The sales people can do sales, and the developers can do development, so, an interesting part about saying, like literally saying "our hierarchy is upside down, management is a support function, it's there to support developers doing their jobs," mean that as you get promoted into management you're going to find that you're the person that has to do all those little clean up jobs and all those little emergencies that crop up, so that the developers can have kind of the luxury of just doing whatever their main multi-month project is, without it

Atwood: I think that's a great way to look at it. And the way I've described it, which I think is similar is, your job as a manager is to throw yourself on grenades to protect me from being interrupted or having to go to these pointless meetings that you have to go to.  And I always told 'em, "I do not envy your job in the slightest.  You're doing the things that I don't want to have to be bothered doing."  I think that understand, the whole, preventing them from being interrupted, is the core lesson.  And I think it's great

Spolsky: It's pretty hard. And it's gonna, you guys are sort of already doing it with StackOverflow, but, I remember the early days of Fog Creek when we were literally two people - it was me and Michael - and we would just take turns doing everything else so that one person could sit and write code.  So, there was a big block of time when I just did literally everything at Fog Creek except for what Michael was doing and Michael was writing, let's see what was he doing, at one point he was writing the store, the online store.  And then at one point I actually went through FogBugz line by line and cleaned it all up, to make it, like, code that would last for a thousand years instead of code that was sort of, kind of able to work (and I think much of that structure still exists).  So, but anyway, the point is that while we were doing this the other person was doing literally everything else.  And one of the things I discovered was that as sort of a small business, when you're operating, when you have all the features of a small business has, including customers, that's at least 50-60% of one person's time, is the overhead of maintaining the fact that you are a business.  I don't even, I'm not sure what stuff is,

Atwood: I termed some of what I do now as "crowd control," basically.  Where, I'm trying to respond to emails and manage the comments and basically deal with my and our online presence.  And that does take some time.  But it's something that hopefully Jarrod and Geoff don't have to worry about.

Spolsky: Well, that's your number one contribution is, so, you know, I would call it, I'd put that in the category of PR basically, it's in some way gaining, making people aware that we have this thing and that we're working on and

Atwood: Well, right.  That's why it's ironic that you left that comment about going dark, because that's pretty much the opposite of everything I do.  I think you just are really chomping at the bit to see the site, which you will be able to.  I'm going to give you the other, secret URL now, which, maybe, someone will figure that one out too, I don't know [laughing] ah, security through obscurity.

Spolsky: [laughing] Oh, OK, then, it's probably something like codinghorror.stackoverflow.com

Atwood: secret.stackoverflow.com, yeah who knows, but somebody will probably write a dictionary attack and break it in like minutes.

Spolsky: Wait, can't I just put together a DNS query and find what all is there.

Atwood: Uh, I don't know if you can, I don't know actually. I mean I know how I set up the prefixes. I dunno, we'll see. I mean, like I said, I don't mind people coming to the site - it's really for your own protection - if people want to see it that's totally cool with me. It's just it's sort of actively hostile to human life at this moment.  So, if you're into that, then, you know, please.

Spolsky: [Laughing] What would be - that's funny - what would be a website that was actually like literally hostile to human life? I mean they could have, well, first of all you'd wanna have flashing because at least you can get the epileptics to go into a seizure. So at a minimum

Atwood: Well, those sites, you're joking, but those sites actually exist. I mean pretty much there's sites that try to game you, they're like "download this video codec to look at this video of, of, who knows what, right?" but it's an EXE which has you know, it's own payload, there's a lot of evil.

Spolsky: And then there's the ones where's it's like, they're like "can you see the ghost in this picture", and you're like staring and staring and you're like "no, where is it?" and then like the whole picture changes to like a screaming person.

Atwood: Yeah, those are, yeah, I've gotten bitten by those before, and then you feel like a doofus. You're like, "I feel like a total idiot now."  So, another thing I want to talk about, briefly is we are using Subversion for source control, which I view as a good default choice. I mean it's free, it's pretty good, there's things that kinda freak me out about Subversion, having used some other source control systems. But it's certainly good.

Spolsky: What freaked you out about it?

Atwood: Well, there's parts of the model that I find unsatisfying. I think it's just about source control in general because I've started to learn a little bit about distributed version control and how that works.

Spolsky: Yeah, Git and Mercurial are the two big ones right now.

Atwood: And really, the key point to take away there is not that, ok, it's this radical new model but they make branching and merging trivially easy. And branching and merging, in Subversion are so painful, both technologically and from a UI standpoint. And that's huge, because I view one of the main benefits of source control is this whole having parallel universes run along side each other where you can explore different techniques. You have a padded room where you can do stuff safely and then merge back into reality once you

Spolsky: Oh wait, you don't have to, so, to have a padded room you don't have to have a branch, you just have to have your own checkout, and you act like you're in a little padded room and then if you don't like what happened then you just delete it and check, you know, go back to what's in the repository.

Atwood: Well, that's right, and you're hitting on one of the key tenants of distributed version control, which is that when you're working "locally," you would still be under version control, just implicitly.

Spolsky: And that's the trouble, while you're in that padded room, you're just on your hard drive.

Atwood: That's right, and see in distributed version control you're actually still under version control at that point. So you have a total record of every little thing that you did. It's a great model, but really the mindset of making branching and merging easy is the thing I object to, the thing that Subversion does not do very well. And I think that the UI like of Tortoise (the file system/shell extension) is not, is still kind of a programmer UI. And I mean that in the negative connotation.

Spolsky: But I mean it is a programming tool. The truth is I find that the UI... Tortoise is a awesome tool, on the other hand the, you're gonna have to learn 4 or 5 important concepts and you're really going to have to get how this thing works before you can start with it.  And tortoise does not relieve you, I don't think any GUI would ever relieve you of that necessity to understand what it is that a source control system is going do for you, until you understand what operations it's doing.  Which is kinda hard, and kinda theoretical, and very abstract in a way.  So it takes a while to "get it."

Atwood: Source control is tough, I mean, it's definitely, I view as the bedrock of software engineering.  And to truly understand source control, particularly once you get into branching and merging and some of the advanced things that you really should be doing - it is not easy to understand that stuff.  'Cause I, at my previous job I basically had to teach that stuff to software developers and it's not easy to teach that stuff to people.

Spolsky: No.

Atwood: And it's not easy to actually even understand it. So you can sort of empathize with developers

Spolsky: confusing it, and so I don't think, yeah, I don't think, I think like Tortoise cannot, like no GUI can make this thing simple for people to understand. Even when you consider just backups.  Think about Time Machine, Apple's Time Machine user interface for backups, which is not exactly source control because it's linear, but it's a little bit like it.  And, even that, that GUI, is barely useful for making it comprehensible to people, but it's still can be kind of a confusing subject.

Atwood: Well, I think there's a way of iteratively doing this, where if you have any skill at UI, and I think a lot of the people, particularly some of the people working on Tortoise, I suspect, have not a lot of skill in that area (no offense to the Tortoise people).  'Cause it is a free tool.  I mean, I'm complaining about things I'm getting for free, which is kinda unfair.  But I feel you can polish it, absolutely.  You have that FAQ model where you say what are the top three issues that people run into using Tortoise or Subversion and you would feed that back into products.  Like, how, from a UI perspective can we ameliorate these problems.  And then you would have that feedback loop.  And I feel they haven't gone very far down that path.  But again, it's free, and it's much better than, say, SourceSafe.  Which I have a whole set of blog entries about please do not use SourceSafe.  So, if you're listening to this and you're using SourceSafe, please explore the alternatives.  'Cause SourceSafe kinda will poison your mind in that it is very, very old source control system and architecturally does not reflect anything remotely resembling modern source control.  Which is really a problem

Spolsky: We should mention.  Tortoise SVN, the person who wrote that code, Stefan Küng, is a guy in Switzerland is a very brilliant programmer, really nice guy, and, it's just all him.  I don't think there's any, you know, people will send in bug fixes and stuff but I think he does, it's pretty much his project.

Atwood: Right.  And, I apologize for coming across as

Spolsky: No, that's ok, it's an awesome tool.  Now, the developers on FogBugz, or Fog Creek in general, recently insisted on switching to Mercurial.  And that's a distributed version control system. Right now the two contenders are Git and Mercurial.  And they are, how can I put it?  They're kinda 1.0 products, and if you thought source code control was complicated, as soon as it becomes distributed you gain new powers but you do so at the cost of extra steps and new abstractions that you have to keep track of.  And it can be even more confusing to use something like distributed source control and to really get it right.  So, there is kind of an uphill battle there, and I think that Subversion is probably reasonable for small projects where you just don't want, maybe they're more casual projects. The thing about Mercurial is it's kinda like unless you do it all the time every day and you're constantly branching and merging for various important reasons and maybe you are physically distributed, like you literally are trying to work offline on a code base and then resynch up later.  Unless you have all those reasons for using it you're probably, if you use it casually, you're going to forget how it works from one time to the next.  So, my experience has been that, if I can spend a day a week working on code - and that's when I'm lucky - I also have to keep referring back to my notes at what the main commands are in Mercurial.  Because, it's just like there are a few too many concepts there for me to keep track of in my head all at once.  And if you do it regularly you should get it, you'll get it, but, yeah.

Atwood: Well, I think there's - you're right, those are new tools - and I think there's a huge opportunity there for someone to come and build a really nice GUI, you know and Apple style GUI around this stuff.  'Cause I still believe with the right GUI you can attack these tough concepts and make 'em easier to break down and comprehend.  Well, with that in mind, I was gonna mention, one tool that helped me tremendously with Subversion was VisualSVN, which is a Visual Studio add-in.  Since I work in Visual Studio mostly it sort of brings me the Subversion experience in VisualStudio so I don't have to tab out to the file system to do operations and I have sort of one click operation to see what pending changes that I've made.  It just really helps me understand the model, 'cause I'm viewing it more from the lens of not "ok, here's a bunch of crap in the file system," but "here's my project in Visual Studio, and here's the current state of it in source control." So, that helped me tremendously. I definitely recommend (if you're in Visual Studio) VisualSVN. People had recommended it, and they were right, it totally changed my attitude...

Spolsky: Didn't there used to be this problem that, that Visual Studio's understanding of source code control systems, Visual Studio assumed that they were checkout/checkin models.

Atwood: Only in 2003. As of 2005, because of Team System, 'cause of the way Team System is a very modern source control system, I mean: love it or hate it, it's definitely modern. In 2005 they changed the model completely. So that was only true of 2003 and earlier.

Spolsky: Cool. For those of you that don't know what we're talking about.  There's basically two philosophies of centralized source code control.  There's the philosophy that is used by Visual SourceSafe, Perforce most of the time, that says, before an RCS, that says before you can work on something you need to check it out and then it's yours and you make your changes and then you check it in again.   And then there's the simpler system that CVS and Subversion use that says if you want to work on something just start working on it, just start editing the file and later you can decide to try to merge your changes in with everybody else's changes but you don't really like "own" or "lock out" the file.  And so, in particular the way the early versions of Visual Studio did their integration I guess, in their first few 17 iterations of the IDE.  The way they integrated it with these source control systems is that they would take care of, like, checking out a file for you, if you start, if you opened a file and started editing it.  They would be like, "alright, let me go check that out for you."  And that is an operation that was really necessary in CVS or Subversion, and so it was hard in the old days to write appropriate plug-ins for Visual Studio for CVS or Subversion.

Atwood: I will say that, as a developer, spend a lot of time learning, really learning, source control, it will serve you as well as anything, any other computer science concept you can learn.  It is really that fundamental.  And I meet so few developers that really get it, I mean I feel like I've, I don't truly get all of it, but I've gone further than most, 'cause I was teaching other developers so I sort of had to learn it in more detail.  But yes, please, if you hear this and you get motivated to learn more.  And it doesn't really matter what you choose, just choose something modern like, here's ones that I like that I've heard good things about, so:

• Subversion definitely, it's free, it's, you know, a default choice for a good reason;
• Perforce, a lot of people really like Perforce;
• Team System, it has a server connected model which can be kind of a bummer which means you have to be always connected, which a lot of people don't, but other than that it actually is very good once you learn that and deal with that and they're changing, I think they're changing the model a little bit to deal with that, a lot of people are saying, "well, I want to work totally disconnected," etc, etc. For some reason, someone in the architecture division decided, early on, that it would be fully connected all the time, they're just basically living with the consequences of that decision from now on. But there are things that are really nice about it, and it's far, far better than, certainly SourceSafe, and I think on par with SubVersion minus the architectural difference of always being connected.

Those are the three that I see always mentioned a lot. And then, of course, the new, hot distributed version control systems, which are Git and Mercurial, are also worth learning and understanding how they work. So, I think those are the key takeaways there. So, before we go too much further, I want to get to the questions, 'cause I don't want to run out of time.

Spolsky: Oh, yeah. Well, there's infinite time, we can grow old together here. Uh, what do we want to talk about, let's, you know what, I'm going to take this one first.

Spolsky: It's the famous walled garden. [Atwood laughs.]