The StackOverflow Podcast

[incomplete]

Intro, advertising

[01:22]

Atwood: We thought the worst case scenario was that we would literally branch the code.

Spolsky: Sure.

Atwood: So then we would have two things to migrate bugfixes to all the time. And one thing - I know you guys use Mercurial, right ?

Spolsky: Yeah, and that would make it a little easier, but I mean it shouldn't really be necessary to branch the code.

Atwood: No, it shouldn't be, but one thing we found was that we did do a branch when we did the big database migration because that was such a big change, we had to have bugfixes go on the main line while we were working on the database change.

Spolsky: Right.

Atwood: Because all the fields change and all the stuff and we did a bunch of refactoring on the database.

Spolsky: Yep.

Atwood: And what we found was merging that in was excruciating - it was so painful. We actually updated to the latest version of Subversion, which is 1.6 I think? And it has "better" support for merging, and it was still just brutal - we missed so many things - there were at least, I want to say, five to ten things that we missed in the merge somehow.

Spolsky: So this is why everybody is abandoning those version control systems that are of the Subversion generation and they're moving to Mercurial and Git, and, to a lesser extent, the other distributed systems. There's something that took me a while to understand about Mercurial, but fundamentally Mercurial is - Mercurial thinks of the world as a list of changes, and Subversion thinks of the world as a list of different versions of all your files. And so for Subversion to figure out where you are on a branch to do the merge, all it can do really is do a diff and say "well, I don't know what happened, but the following things - we're different from you guys in the following ways", whereas Mercurial has this sort of added info of all the steps that were taken and all the transformations that were done, so it basically has more information to use in the merge and making the merge successful.

Atwood: Yeah. I think that's the key phrase, is that it feels like Subversion does so little to make your merge successful that the success of the merge relies entirely on you doing every stinking little thing correctly, and if you miss anything then you get a bad merge, and the thing that sucks is that you don't figure out until a week later that this bugfix that you put in is not in the codebase.

Spolsky: Yeah, because it might compile.

Atwood: It pissed me off - every time I ran into I just got very very angry because I felt like my tooling had failed me.

Spolsky: Well I would highly recommend just switching to Mercurial, also because you guys are all a distributed team, right? Like the three of you are all in different places, and we're going to want to have the hosted version that FogCreek is doing and it would be really good, even if we have to fork the code, for us both to be using Mercurial, because then we could still throw each other bugfixes, even if we forked, you know, years later we can be throwing each other bugfixes.

Atwood: Are you like announcing that now on the podcast ?

Spolsky: I guess I might as well, what the heck, I've waited - we have no secrets. We haven't even hired somebody to develop this thing yet; Hey, if you are a StackOverflow listener and you're a really good developer, and you're pretty good, especially - we'll hire you even if you don't necessarily know ASP.NET, and ASP.NET MVC, but if you do that's a big help, and you want to work on Stack Overflow, then we've got an opening here at FogCreek, so email your resume to jobs@fogcreek.com.

Atwood: But you have to be local, right ?

Spolsky: Nope, we'll hire - you just have to be, you just have to have permanent right to work in the United States, so that means that you already have permanent residency like a green card or US or Canadian citizen. A US or Canadian or Mexican citizen with a Bachelor's in Engineering. I don't know, it's complicated; basically, no H1B's, because you just can't get them anymore.

Atwood: Why don't you, OK, well that's cool, that's awesome that you're opening it up to the whole United States.

Spolsky: We always do, about half the people we hire we wind up moving to New York, more and more actually, I can't even think of any local New Yorkers that we've hired.

Atwood: OK, cool, so there's a relocation sort of thing there ?

Spolsky: Yeah, you'd get the whole package: FogCreek stock, the relocation package, the free lunches, the uh, anyway.

Atwood: Cool. Well that's exciting, because I'll tell you my mailbox I don't want to say overflowing, but every few days I get an email from somebody that wants, would like to use the Stack Overflow engine in some way, and I just don't really have a good answer for them, so this is exciting for me because now I have at least a reasonable answer for them, which is that we're working on a hosted version of it that's going to be run on the FogCreek infrastructure.

Spolsky: Some hypothetical person that we really want to hire hopefully will hear this podcast and send me a resume.

Atwood: The funny thing about that is that we're very excited about that and we feel that it fills a real need. It's a service that you guys already do very well - it's a good fit on every possible level, but the thing that makes us all pause on the team is that other people are going to look at our code and we're like "ooh, is our code really good enough for another developer to come in and not tear their hair out?" I mean, we're probably being oversensitive, but, I think that's the reaction I have with all my code - is it good enough for other people to work on? Maybe you should spend a month refactoring it first.

Spolsky: Well, I think everybody feels that way, and that is actually true that there is - there's kind of two levels of code: there's the level where it runs and it's debugged and you're kinda happy with it and you can continue to work on it if you need to. Let's say there's three levels, that's the middle level. The bad level is you know it's bad and it's a pain to work on and any time you want to change something you know you're going to be pulling out your own hair, and then the top level is like you could publish this in a book because after you got it working you went over it and refactored it seventeen times and cleaned it up and did all kinds of extra work that didn't get you any extra functionality, but did make it code that anybody could dive into, so maybe you've renamed things, you've cleaned things up, you've reorganized things several times, you've gone through the code trying to make it like literary code where the comments just smoothly, seamlessly flow with the code so you can figure out what's going on.

Atwood: Our challenge, too, is that we started in one of the betas of ASP.NET MVC and we sort of learned, not that MVC is super-complicated, but we learned MVC as we were going.

Spolsky: That happens so often.

Atwood: And then there's points in the code where we did things that I don't think really - they're not wrong, they're not horrible, I don't think anything in our code would make you question our core competency as software developers, I will say that, because there's always The Daily WTF code where these people shouldn't even be near a computer, much less writing code. I don't think we're at that level, but there's definitely parts where we did things that aren't in the spirit of MVC because we didn't really understand or, honestly, some of the pieces of MVC weren't there when we started. All we had was the root pieces of it and they sort of built up pieces around it. For example, validation - there was no real form validation framework so you had to sort of roll your own, and that's essentially what we did we rolled our own validation. If you were starting out now there's no way you would do that, but we have all this existing code that we have to support.

Spolsky: So there's no reason to change to their new, modern, official form validation.

Atwood: I'm a big fan of refactoring, believe me, and when I started I told Jarrod, one of the first things I told him was "look, we're going to rewrite this app three times, just top to bottom", that's just the way software works.

Spolsky: You mean you're going to start from scratch ?

Atwood: Well not literally, I was being a little dramatic for effect, my point is that you want to refactor all the time, that's part of having healthy codebases, you go in and you just tear things up that aren't good and make them good, right? Even if you don't - it's a question of balancing resources, you don't do it all the time, but you periodically tear.

Spolsky: You take a piece and tear it out, or you can take a piece and just try to clean it up.

Atwood: Exactly, you don't just say "OK, I'm going to fix this" - it's kind of like working with legacy code, you're like "oh, I'm going to touch this as little as possible, just to get it to do this one thing that I want then I'm not going to, I don't want to break it". I don't want to have that attitude, that walking on eggshells attitude towards code because I feel like that's unhealthy. At that point you have a legacy codebase and you're not really alive as a project in my opinion, so whenever we do stuff I encourage the guys to - and they do it all the time - to go in and just rewrite things, little things, that are not the way they should be to make them more elegant or just not melt-your-brain when you look at them. So I'm for that, but on the other hand, it is true, and we do this all the time, we regress. Every time you touch code, every time you change something, the odds of you breaking stuff are, you know, really high.

Spolsky: So that is what the trendy boys would call a "code smell".

Atwood: Yes, yes, yes. And then of course you could that's what TDD is all about, right? Theoretically it's like getting a whole set of unit tests in place so every time you change something, you know, you're protected, you know that nothing's going to break.

Spolsky: And I thought the theory, I mean it sounds like the theory behind some of the SOLID principles, which we like to butcher on our show, had something to do with making sure your code is in a place where every time you change something you don't break all kinds of other, unrelated things, because if you follow these principles presumably every time you want to change something there's, you know, one elegant file that you just have to find it and go in there and change a zero to a one and everything has been thought out for you already and so it magically happens without any bugs.

Atwood: Right, and before people, and true, we could, we do have some unit tests, we could have more unit tests, but I find too that some of the regressions you get I don't know how in the world you would have a unit test that would catch them, like the other day I went in and I normalized one of our CSS classes that had to do with the OpenID field, the way it displays, and made it the same across, we were using different things in different places which didn't make sense, so I normalized it. This actually broke some of the JavaScript on the client in a very minor way, nothing horrible, but the login page JavaScript does some sort of stuff to help you figure out what OpenID provider you're using.

Spolsky: I was having trouble logging in.

Atwood: Yeah, well we did some login refactoring and broke a few things, nothing serious, but minor, annoying things. This was one of them.

Spolsky: Was it, wait, tell me Jeff real quick because maybe this is another bug that I need to file, like I was clicking on the little MyOpenID thing and I was typing my name and I was clicking in some other place and it was just failing to make it look like I was logged in but if I'd actually, like it was telling me "no, that's, you're wrong, you cannot log in", but actually if I went back to the site then I was logged in, is that that bug ?

[11:48]

....

[17:16]

Spolsky: That's cool, and you know we were talking earlier - I wanted to mention something that occured to me - we were talking about how you guys were learning ASP.NET and MVC as you went along, and I can't remember a time when I wasn't building a new application in an environment that I did not have any experience with; whether it was a programming language or the frameworks, it seems like you're always learning as you go along for some reason. You never really get to build a new application in a framework that you know really well.

And I was just trying to figure out why that was. One of the most famous things about the Juno codebase was that the guy who started writing the code was learning C++ for the first time, and decided that he needed to write his own string class. And the main reason was .. I don't know. It was admittedly a long time ago. STL was not yet standardised. I'm pretty sure it existed but it was not well known - way before Boost. And Microsoft had a certain thing called CString in the Microsoft Foundation Classes but we needed code that could run on a UNIX server and on Windows. So he said, alright I'm going to make my own string class, and he proceeded to invent a string class that made every single possible mistake that you can possibly make in the design of a string class in C++.

It's funny 'cos there was this book called Effective C++ by Scott Meyers, did you ever read it that book, Effective C++?

Atwood: Are you kidding me? Is this a joke? Continue.

Spolsky: No. Sorry. It's a book that basically says, C++ is enough rope to hang yourself, and then a couple of extra miles of rope, and then a couple of suicide pills that are disguised as M&Ms...

Atwood: <laughing> Disguised as vitamins

Spolsky: <laughing> Peanut M&Ms, and you know, and the occasional electric knife and so forth. And there were a couple of books by Scott Meyers in the heyday of C++ called Effective C++, and there's one called More Effective C++ and they're great books, and they basically show you, look you're going to write a class and you're going to do this particular thing for your copy operators, you're going to make all these mistakes, and here's why these things are mistakes and here's why these things are going to blow up in your face. And you're not going to find out  until the Space Shuttle explodes. And so don't do any of these things.

And he had, I don't remember, but there were probably 50 different things in each book that you shouldn't do, and the easiest way to learn this would be to take the string class we were using in Juno which literally made every one of these mistakes. And it was fair because he was learning C++ as he was going along, the guy who wrote this code, and these are all the kinds of things, when you read Effective C++, even if you were a pretty good C++ programmer, this was just eye-opening because you never thought, wow I never thought that if I did that that it would have this other bad implication.So it's kind of a dangerous language, because if you haven't read that book then you make all these dangerous classes.

[19:59]

Spolsky:  It occured to me what happens.. how often when you are working as a professional developer how often do you start a major new project thats going to have a long life expectancy.  Like here at Fogcreek we've launched fogbugz which was already legacy code, City Desk, CoPilot - a lot of that code came from VNC.  So maybe it has twice happened that somebody has sat down to write a major new piece of code thats going to have a long legacy and thats in the 8 years that this company has been around, almost 9 years.  It just doesnt happen that often in your carear that you get to start over.  When you get to start over, lets set it happens every three or four year to a programmer  that you're actually starting to build a new gigantic thing, heck every year.  You're going to look around and you're going to see whats the best possible framework you can develop this in, because  I'm going to be living with this for a long time, and you're going to evaluate the frameworks and what programming languages are available to you, whats the best tool for the job and thats awesome.  When you make that decision  the chances are its going to be a different framework than what was the best framework 4 years ago when you made the last decision.  Right?  Its just reasonable that you'll be looking around

[47:52]

Spolsky:  Yeah, we have another question from a listener called Tom. 

Tom from Manhattan: Hi Joel and Jeff, this is Tom from Manhattan. I believe the phrase to describe me is "first time, long time". My question is: What steps, if any, did you take to protect the intellectual property built into StackOverflow? I'm working on a start-up of my own and want to know how to protect myself from someone producing an exact copy of my site. Or, in general, whether it is even worth worrying about at all. [...] more on the ball. What would stop them from copying the best parts of StackOverflow before you guys got a critical mass of users?

Spolsky: Too late. We already got the critical mass. Ha!

Atwood: I think a lot of the concern about copying is sort of misplaced. I think you want to, I don't know, I'm more of the mind that: first of all create, if you've created something worth copying at all, which is already kind of rare, so 90% of the time when you create something and nobody cares. It's not going to get copied so it's all academic. It doesn't really matter how many protections you have in place because nobody cares.

Spolsky: Yep.

Atwood: So at the point which people care enough to make a copy of what you're doing, first of all it's a huge compliment. It means you've done something good that's worth copying. And I think the way to stay ahead in the whole are people going to copy me race... first of all, it's a weird negative mentality, I don't like fear-based development. I like to develop things because I think the world needs them or I think they're cool or I think they would be interesting and help people and not because I'm afraid other people are going to copy me or what I'm doing. And I think you want to continually evolve. I mean if you look at, at even the iPod for example, how many people have tried to clone the iPod now?

Spolsky: Yeah, by the time they come up with their clone they're a couple of generations behind Apple anyway.

Atwood: That, that's exactly my point. A living software project is continually changing and is very, very hard to copy for that reason. At the point which you're locked in for a year and your site hasn't changed for a year, then you should start to worry that people are going copy you lock stock and barrel. But as long as you're continuing to evolve you're just going to continue to get better and it's going to be really hard for people to copy you. And certainly that's the intention on StackOverflow is that we're a living software project. We're going to continue to add features, refine our site, make it better, make it cooler. That's my theory.

Spolsky: One way to evolve: If you're in this particular niche, if you're product happens to be the type of product that benefits from having a larger number of users, and there's a certain class of website software business that the more people it has, the better it gets and it's called a Network Effect. Because the bigger the network, the number of possible connections in the network is order of n-squared and so the more people you have the better it gets. 

So think of auction sites, eBay just totally won the online auction thing because there were more people looking at auctions and there were more people selling things so it was just a more competitive market. And similarly, Twitter has pretty much owned the space of whatever you call that thing, the Twitting, Twittering... space because there are so many people on there and you could go make a new and it could have 10x as much functionality but if there's nobody there then it doesn't matter. And so I actually, that's the Network Effects thing and it's not even necessarily the case that you have to keep ahead in terms of what features you have, it may be just enough that you capture that initial, critical mass of people so that you are always the better place. You know, if you have a very very rare question about programming you're going to want to ask it on StackOverflow because StackOverflow has 3 million unique visitors a month and there's just a lot more people that are going to look at it on StackOverflow than on other sites or on anyone else that might come up and try and compete with us. 

And in fact I have like a whole theory on this and the theory says: If you start a business and you're worried about people copying it, stop worrying about that because what you'll actually find is that if you're doing something that's really going to work, that when you try to tell people about it they won't quite get it. You'll tell them "I'm building this thing and it's going to do blah blah blah" and there are two possibilities at this point: either it already exists because it's kind of obvious that you can do a thing like this (e.g. bug tracking software: "Oh yeah, there's a bunch of sites that have that, there's a bunch of companies that make that") or it sounds in some way like it shouldn't work and there's something that makes the people hearing that idea make them think it's not going to work and not really want to do it. And those are the super exciting sites. The super exciting sites are the ones that no matter how many people you tell about it they're not quite getting why it should work. 

For example people would tell me about eBay for months before I finally tried it and I would just say "I don't get it. You just send money to some person you just found on the Internet? Why wouldn't everyone just pretend to have laptops for sale and just collect on the checks and just disappear"? And so, I really fundamentally thought eBay wouldn't work and indeed and while eBay was growing and gaining that critical mass and becoming this fantastically successful auction site, which is now full of scammers but whatever, I was sitting there saying "ah, couldn't possibly work" and if I had been interested in this marketplace by the time I had gotten in there it would have been too late: they would have already had critical mass. 

You know, so I'm actually looking at StackOverflow and one thing I'm noticing is that the site that we consider to be one of our main competitors, which is a for-pay site that pretends to not be a for-pay site and kind of cloaks itself and stuff and we're going to call the hyphen site from now on because they have a hyphen in their URL. They still really haven't changed anything about the way that they do business and we've been out now for 6 months now and we've taking away traffic from them at a fantastic rate. Well, we don't actually know if we're taking traffic away from them but certainly taking market share away from them at a fantastic rate. And they must be aware that we exist and they haven't changed anything about the way they do business and someday, when someone writes the history of StackOverflow in Wikipedia, they're going to say "It took that hyphen site 2 years and 4 months to even respond to StackOverflow" and to do something and to, you know, respond to the new competitive challenge. Because somehow people used to their way of doing business and are kind of locked into it and if you've got something genuinely new they're not going to notice and change their way of doing things for, for years. 

[54:16]

...

[63:58 ends]

Outro, advertising

[65:15]