The StackOverflow Podcast

[32:40 - Joel saying he doesn't think it's that hard to run a DNS server]

Atwood: Well, I just think it's more important than you made it out to be... I don't know, I think I'm with Google on this one. What I've learned is that DNS speed is actually kind of relevant.

Spolsky: Yeah. No, it's definitely relevant, I just don't think it's that hard to put up your own server that's pretty fast, that does DNS pretty well.

Atwood: Yeah.

Spolsky: But sure, go ahead, yeah fine, outsource it. Maybe I'm wrong there; maybe it's worth outsourcing.

Atwood: So I've got a question for you: did you pick a listener question? We promised T-shirts...

Spolsky: OK. You know, nobody listens to our podcast any more. Because last week we said, whatever the best question was, is going to get a T-shirt. And we got two questions. Two! That's it!

Atwood: Well, they have a 50/50 chance of winning.

Spolsky: That's right. Well, I'm going to play them both. Listeners, use mental telepathy. Think in your head who you want to win, which one you think is the better question, alright, here's the first one.

[Listener question]: Hi Jeff and Joel, my name is Kelly French and I've been listening to your podcast and enjoyed it a lot. In the spirit of stackoverflow, I want to offer an answer to one of your questions before I ask mine. When you asked: "Why do you have to call Dell to get a better price?" it's a matter of control. They want to make sure that their competitors can't scrape their website and then offer a lower price and say they beat Dell's price.

[Listener question, cont.]: So, my question is: if it's so well-known that some programmers are ten times better than other programmers, why don't employers offer ten times more salary? Let's say they did: what would be the effect if some programmers actually did earn ten times what a starting programmer did? E.g. an intern was like $25,000 and a rockstar could earn $250,000, just as a regular job, not as equity or starting their own company. Thanks a lot, and I'll keep listening.

Atwood: Well, I like that one. I think that's a really interesting thought experiment. But let's listen to number two.

Spolsky: Alright. Well, do we want to answer this one first?

Atwood: I thought we were going to listen to both questions? Now I want to hear the other question!

Spolsky: Alright, well let's play the other question. It's a umm... here we go. Uh-oh.

[Joel struggles with Audacity for a few minutes; Jeff makes bad jokes at his expense]

[35:45]

[Listener question]: Hey Jeff and Joel. What is your opinion about developers creating databases? I'm a programmer at a huge company, where everyone has a narrowly defined role. A few days ago a friend in another department said that "developers shouldn't make databases", meaning that it's something that only a data architect has enough competence for. I'm no Sid Meier, but I've been programming since middle school; I have an MS in software engineering and a dozen years experience. I think I'm qualified to create a database. It sure doesn't seem like rocket science, is there something I'm missing? What do you guys think?

Atwood: Well, I like both of these questions. I think both of these guys should get T-shirts.

Spolsky: Alright. Two T-shirts. So, Brad. I have Brad's email address so I'll get in touch with you and get you a T-shirt, and the other person... Kelly? I don't have yours because you just called in.

Atwood: I'm sure once he hears the podcast...

Spolsky: Once you listen to the podcast, could you send an email please, to podcast@stackoverflow.com saying "I'm the guy that called in" and I'll get you a T-shirt.

Atwood: So OK, let's go with the first question which was thought experiment...

Spolsky: Sure, thought experiment. If programmers really do vary ten times, why don't they make a factor of ten salary?

Atwood: Well, immediately there's a problem, because I can't think of any other job where that actually happens. I mean, the only place where you have these huge salary disparities in the real world is like CEO, right? CEO's make stupid amounts of money.

Spolsky: Yeah, well, that's just because of corruption. We're corrupt.

Atwood: Yeah, I mean that's crazy. I mean if you take that one crazy example off the table, is there anywhere else that this actually happens?

Spolsky: Yes.

Atwood: Really?

Spolsky: Sure. Let's really do the thought experiment. His question was should an entry level programmer make $25,000 -that's kind of low in the United States-...

Atwood: "Kind of low?" That's extraordinarily low.

Spolsky: Yeah, well, but I'll bet you there's a place where you could find someone who's an entry level programmer doing very boring stuff at an insurance type company in a very inexpensive part of the country to live in. You know, like they're doing RPG-2(?) reports for the government in Alaska, and their making $25,000. Or in the military, something like that.

Atwood: OK. So where's your example of where this actually happens? Where you have these massive disparities?

Spolsky: And I'll bet you that there is... I mean, how much do you think that Guido van Rossum takes home a year? From Google? Do you think he makes $250,000?

Atwood: Oh gosh - I have no idea.

Spolsky: I think he probably makes, and this is just a guess...

Atwood: But do you think he makes ten times as much as the average engineer at Google?

Spolsky: Not the average, the low.

Atwood: The lowest. But Google doesn't have lowest! They only hire the top engineers.

Spolsky: No. The $25,000 guy in Anchorage, and Guido van Rossum will be making $250,000, and there's a ten-to-one order of magnitude there.

Atwood: Oh, I see. So you're extending it outside one company. I was thinking within one company.

Spolsky: Well, within one company you don't really have, I mean, that's because Google is pretty strict about who they hire. And I don't think that companies are generally... like if a company is even aware that there's such a gap between the good programmers and the bad progammers, they're not going to want any of the bad programmers around at all. Because the real problem is not that the low-end -I don't want to say $25,000 programmer, I want to say the least-productive programmer- the problem is that it's not that they're one tenth as productive as Guido. The problem is that they're negative productive. Like, massively negative productive.

Atwood: OK. I think you've interpreted this in a strange way. I think you're correct, in what you're saying, but I think it's a little strange interpretation. Because you're right, Guido van Rossum probably makes ten times as much as grunt programmer in a very rural area of the United States where it doesn't cost that much to live. But I think the spirit of the question was more like: you're working at a company. In any given company -like the typical company, not necessarily Fog Creek or Google or even Microsoft- you have certain programmers that are much more efficient than other programmers. So, what would it be like if "Programmer A" made three times as much as "Programmer B" but that programmer was also three times more efficient. The immediate problem is that I think you'd have a riot. If this ever got out, that there was this disparity in pay, you would basically have a riot. But I don't think that you could have a company like that.

Spolsky: I think that a lot of companies do that. A lot of companies just sort of secretly give people big bonuses. And they're paying them under the table and they're like: "Listen, I'm going to give you a little bit more, but don't tell anyone." They probably pay people... I think most people feel like they should be paying their good programmers more than their mediocre programmers.

Atwood: But it's never on the order of three times as much.

Spolsky: No, you're right, it couldn't be that much.

Atwood: The people who tend to excel tend to make, I don't know, 10% more, 15%, maybe 20% more.

Spolsky: I know, it's a bargain, those are the bargains. Those people are bargains, to hire those people.

Atwood: But I think if you actually had a company where you had four to five times multipliers, in salary, I think a) word would get out, and b) you would not have a company very much longer.

Spolsky: Don't forget if you have a programmer who you have three to four times multipliers in productivity inside the same company, those highly productive people are going to be really disgruntled. Because they know that they're carrying all the water. And eventually they're going to leave, I think. I think they're going to tend to migrate out to other companies where they're more appreciated than other people. You know where they're going to go, to startups or...

Atwood: I think that's what I'm eventually coming to. I think that what you have is this equilibrium inside companies, where all the good programmers tend to go to company A, and all the not-so-productive programmers tend to go to company B. And hence you don't have that much variance around the norm, because you can't. If someone's really, really good, you're right - why would they hang around when everybody else is not as good as they are, if they're a huge outlier. Or if you're really, really bad eventually you're going to get pushed out. You're going tend to reach an equilibrium, and I think that's just the natural state of the world. I don't think that what he's describing, this idea that you would actually pay people four to five times as much, or even ten times as much, is at all sustainable. I just don't you could even do it. It wouldn't work.

Spolsky: And don't forget also that the claim that good programmers are ten times more productive than the bad programmers or whatever does not mean that there's an equal distribution. It mean that there exist some programmers who are just amazingly productive, and those programmers may be the kind of people who write operating systems in their spare time which become enormously successful and the third most popular operating system in the world. They may be the kind of people who invent new languages and new compilers, who start companies and invent whole new ways of programming web applications in Ruby, and whatnot. Those are those highly super-productive programmers. The other thing about the productivity which is kind of interesting is that a lot of times, I've seen that productivity comes from just doing something smarter. So, you're not really working twice as hard. You don't appear to be working twice as hard, you don't appear to be writing twice as many lines of code, you don't appear to be generating twice as much code. But you figure out enormous shortcuts that make things work; so, you are vastly more productive, but not because it looks like you're lifting twice as much weight as anybody else. And then, on the other side, the people who are unproductive are sometimes unproductive in some very disguised ways. Like it's not like they don't come in to work, they still come in to work. It's not like they're not checking things in, they're checking things in. It's just that they have a lot of bugs, or it takes them a really long time to do something, but they stil do it and they still appear to be working until 9PM, so they look kind of productive. So there's all kinds of ways in which the people who are highly productive may be productive in kind of a disguised way. I remember one case in which I got something done incredibly productively because I realised there was this gigantic shortcut where we could reuse a bunch of existing code. So it took me ten lines of code to do what should have been... what other people were doing with 100,000 where they were rewriting a bunch of code. And I was like "Well, let's just use that old code." That's an example of being unbelievably productive, but not in a way that anybody's going to give you any big bonus for. They aren't going to say "Oh, well, I'm going to pay you a hundred times as much because you figured out a[n] order of one hundred shortcut." They're just like: "Oh, that's a clever shortcut, you're bright." They don't really see the productivity advantages of that per se.

Atwood: So the summary is that the world is just not fair, in general. I mean, there's no way the market could...

Spolsky: I think that eventually it does though. I think that what happens is that the really bright people wind up at the really good companies and are really successful, and they make a whole bunch of money on the stock options. Not always, I mean there are probably a million cases in which someone didn't get what they had coming to them. But a lot of times those really good and really productive programmers do wind up getting paid for it, because they get the better jobs.

Atwood: Well, I think the difference is, you have to be internally motivated to recognise that you are good, and that you should be paid what you're worth. And honestly I'm not sure that most programmers are actually good at that. Even the ones that are really good, they're sort of self-effacing, I mean they're not willing to push in the way that they need to push in order to really get what they're worth. Something to really think about, maybe that's the one thing to take away from this question: if you really feel that you are good, legitimately good programmer, and you're not getting what you're worth, you have to push to get what you want. I mean, nobody's going to hand it to you on a silver platter. You have to start a company -that's the extreme, that's the Paul Graham route, you're awesome so just start a company. Or b) seek out other companies where there are really good programmers, and they tend to pay programmers a lot because they recgnise the value of a good programmer. And that's the only way that's really going to work out for you. Nobody's going to come up to you magically one day and say: "You know what, you are really good, you deserve to make three times as much as these crappy programmers that you work with. That is never going to happen. So you've got to get out there and push a little.

Spolsky: Yep.

Atwood: So, question number two. About creating...

Spolsky: Question number two, "can developers do databases?" asks Brad in Minneapolis.

Atwood: I have a weird opinion on this which is that I think developers should where many hats. I think you've got to be a generalist to be a good developer but not everybody agrees with that. And I think that certainly at some organisations they want to enforce these walls. Because you have these little fiefdoms of "We do data, you guys do programming." [mocking] "If you start doing data, then you're encroaching upon our kingdom, and we can't have that."

Spolsky: Yeah, because that's really what it is, actually, is a lot of... what's the word I'm looking for, dumb companies? No. A lot of companies... yeah, fiefdoms, maybe. I once met a database developer who actually had an associate degree, an AA degree, a two year college degree, in database design and development, as in, being a database developer. And I kind of thought that was weird, I mean isn't there just you... make some tables and some columns and stuff like that, what exactly are these database developers learning over there? And I guess they're learning how to normalise databases, and... So I mean there is obviously a whole body of knowledge around how to make a database work really well.

Atwood: And let's take the opposing position as well. I've certainly seen developers make really bad data decisions because they didn't really know what they were doing.

Spolsky: And that stuff is hard. I mean Brent Ozar helped us with our database in so many ways. Taking really good developers, and they still don't know half of what a good database developer knows about how to make a database highly performant, at least, and highly efficient. I think that is true. I think that the larger the organisation the more benefit they get from having specialists that really understand certain areas in depth. And having generalists who just kind of know a bunch of things, that's useful for getting started, but at some point there's just too much to know. If you had to know everything there was to know about ASP.NET, and everything there was to know about C#, and everything there was to know about your database engine (let's say it's SQL server), that's three really, really large domains. It's possible that you could be good enough at all of them, but it's more likely that you're not; you just haven't had the time to really become deeply knowledgable about any one of those three areas.

Atwood: You know what, I'm, wait, wait, I'm kind of having flashbacks to your "Developers must know C" argument. I mean if developers don't know just the fundamentals of databases and indexing and just database performance, I mean this to me is your pointer argument. It's like "You have to know this stuff." I mean, you're going to write code that just sucks if you don't understand this stuff.

Spolsky: Well, you might have a database developer. If you have a database developer working with you who can do that stuff for you then you don't have to.

Atwood: But it's just not the same as knowing it versus someone telling it to you.

Spolsky: Yeah, you're right.

Atwood: At some point you eventually learn it. I mean, unless you're a brand new developer who's never worked with a database. And how does that even happen? How do you even get there?

Spolsky: And now I'm going to insult a bunch of people which is what I always do on this podcast at about this point in time, but... even though there's a strong argument to be made for the benefit of specialisation and having some members of your team knowing the database really well, realistically, it is often the case that the database developers and the database administrators, I don't know why, but realistically sometimes they don't even know as much about how to develop for a database as the programmers do. So in a lot of these organisations that are forcing this particular distincition, where they're like "No, no, don't work on that, we have a database developer for that part," and the database developer, frankly, is somebody that wasn't quite qualified to be a programmer. And so, they're not even as good at the database development part as the programmer, the main developer would have been. I'm not saying that's always the case, or that it's necessarily the case, but I'm just saying that I've seen that way too many times, where the database developers are often failed generalist developers, that were just moved into database development because it was just easier or there was just less stuff they had to figure out.

Atwood: To me, the worst thing you can do to a team is take away control of their destiny. Where you say, "OK, you're creating a product, some software product, that involves a database, and you're not going to have control of the database," which is a huge part of your application. So immediately, if I was working on that project, my affinity, the amount I care about that project just went down dramatically, because you've taken away my control of my own destiny. I want to create something really good; and you have to believe that most of the people working want to create things that they're proud of. They really do. I mean why else would they have a job? What's the point of working? Creating things you're proud of. So they're going to do everything they can to make it a great product, they're going to try to make it succeed. So, if you take away their control...

Spolsky: [mocking Jeff] Ahh... what an enlightened and naive perspective. Think of all the people working at crappy places that don't give a ****, they just want to go home at 5 o'clock, nobody care about what they're working on, they don't care what they're working on, they just want to go home. And now they've been told that they can finish their whole damned project by Friday, but they've been told that they're not allowed to work on any of the database stuff, because that has to be done by the database developer. And the database developer is saying: "Well, this is going to take me six months to do those thirty seven tables that you need!" And you're like: "I've already made the tables, they're already there." And he says: "No, but you haven't done this as a database developer, this is unacceptable, this is crossing lines of authority, blablablah." This is just like unions all over again - they want to shout at you because you've turned on a light or something.

Atwood: I've worked at large companies where I've had this experience and we've never had a good interaction with the database group. They've always just interfered; it wasn't their baby, they don't care about it as much as you do. To them, it's just another database that they have to set up. They don't really give a damn about it. Whereas to us it's a core part of our application, we care intimately about it.

Spolsky: I've got to tell you... a major investment bank which was a large user of a product that has Fog and Bugz in the name. This major investment bank had called us up because they had FogBugz running, and it was running fine, and everything was just perfect and peachy, and they had decided for some reason to try to outsource things. They didn't want to outsource everything, because they needed the in-house people because those people were smart, but they thought it might be a really good idea to outsource to some external company in India at least their databases. So they took a perfectly good, working SQL server database that was running fine and everything was happy and nothing was broken about it, and they were now trying to move this database from a server in New York (where the bank was) to a server in London where it would be managed by a group of so-called database administrators in India. So, I was on the phone with our customer, who was in London, or New York, or something like that. We were trying to get this thing to work, and he wasn't allowed to touch any of the buttons in SQL Server Administrator. So he was calling this outsourcing company that was now responsible for SQLServer, trying to get it to work, and we called up this guy in India who didn't know what he was doing. He had no idea how to operate SQL Server Enterprise Manager at all. He knew how to do the most basic things, like restore and backup and that kind of stuff, but he didn't know how to do into any of the dialogue boxes and change any of the settings, this was all completely foreign to him. So at some point, we were just: "Could you please just give us access to SQL Server so that we can just do this?" because we knew how to do it. I knew how to do it, he knew how to do it, eventually we wound up on this gigantic conference call where we're literally providing tech support for this so called outsourcing database company in India. We're sitting there saying: "Right-click on such and such; choose properties. What do you see there? No, read what's on the screen. No, just read me the words that are on the screen." You know, that kind of tech support call?

Atwood: This is the nightmare scenario, this is just so painful to hear.

Spolsky: Yes. This is supposed to be the expert. Basically what happened is 24 hours of work from various high paid people, trying to get some checkbox checked in SQL Server Enterprise Manager, by some person in India. And presumably the whole value of this operation... the theory is supposed to be able to outsource some simple part of your IT operation to some company that's going to do it. What you've really done is that you've just bungled everything, and you've just made it really difficult. You've just had person who you're now paying to stand in between you and the dialogue box that you need to get to to make that change. The whole thing was like a hysterical story, like a comedy of errors.

Atwood: It's depressing. It's not really funny to me.

Spolsky: Right. And the funny thing is that you just know that they never fixed this. Everybody probably knew that this was broken, and a waste of time and a waste money, and just broken in a very particular way. And it didn't matter, because this is just the way that large corporations... this is what they do. They're like: "Well, we've got to outsource something, let's outsource the simplest thing, let's just outsource the administration of SQL server."

Atwood: Let me something nice about developers, because I've been criticised for stereotyping developers in certain ways on recent podcasts. And I think those stereotypes are largely true, but software developers are very good at being self directed. You give them a mission and just get out of their way. Almost all developers with the exception of the really bad ones that are the bottom of the barrel, the bottom 10%, that shouldn't really even be working as software developers, I think are very good at this. If you give them a mission and just get out of their way, and give them all the tools they need, and don't like inhibit them by saying "Oh, you've got to go through our database group", they actually will deliver, it's what they do. They really enjoy just wholescale building of 'stuff'. Sometimes they build too much stuff, that's actually the failure condition, they'll build this grandiose castle that does way more than you want it to do, ironically. So you actually have to gate them the other way, which is: "Don't build too much." Just give them a mission and just get out of their way is the best possible way to manage developers that - at least the ones that I've worked with, most of them.

Spolsky: The good ones, the ones that are the "get things done" part of "Smart and gets things done".

Atwood: The more you put in front of people, "Oh, you've got to have the database guy, you've got to have this guy..." It's just unnecessary, it's just blocking stuff that going to keep them from doing their best work.

Spolsky: Yeah, those are just ways of basically breaking a functional team, demoralising people that were... it's so sad.

Atwood: It is a bummer. I mean in the extreme case, let's get back to the question, if this is an ongoing thing where you can't meet someone enlightened who sees that you know what you're doing, that you don't need to be babysat and have a database guy guiding you, then you are kind of at the wrong place, and you need to be looking at alternatives. Maybe you should go to careers.stackoverflow.com ?

Spolsky: That's a good choice; OK. I want to do a Stack Overflow questions.

Atwood: OK, do it.

Spolsky: It's question 1867857.

Atwood: Say it again, 186...

Spolsky: 1867857. And the question is, have you even restricted yourself to using a subset of language features.

[57:02]

[1:07:19]

Atwood: We also have a transcript wiki which will be linked from the show notes where people who can't listen to these podcasts can actually go on line to see select parts transcribed... that are interesting .... um ... and anything else Joel, did I miss anything? 

Joel: Um.... no. See you next week.

Atwood: See you next week.