The StackOverflow Podcast

Atwood: Nope, just right into the content, man. That's how we roll.

Spolsky: Alright, well most of our questions here are about StackOverflow as a matter of fact. So let's just start throwin' them out there.

[2:21]

Christopher Leary: Hey Jeff and Joel, I've been really interested in the development of StackOverflow's Karma system since I've been trying to come up with one of my own. You guys seem to assume that there exists a mapping of actions to Karmic weights that will steer the whole userbase in the right direction in a way that's also going to scale.  I've read the transcript of Shirky's "A group is it's own worst enemy" talk, which made me wonder if this is even possible?  Do you really think that a theoretical greatest possible weighting system would be good enough to deter the pathological users and encourage everybody else?

Spolsky: Yes.

That's the whole question.  Go ahead Jeff.

Atwood: I think the system we came up with - I mean I don't pretend to have an idea of what the end state is for our system.  I actually don't know, I mean because there's going to be continued growth in reputation.

Spolsky: Yeah.  It might have to change.

Atwood: I think one thing we changed - one thing we did early on that I think was hugely important and we should have done it earlier - was put a cap on how much reputation you can get in any given day.  And I was just looking at a request on UserVoice that someone thinks we should also have a cap on the amount of reputation you can get from any one response.  Jon Skeet was noting that he felt it was a little unfair that he got 46 up votes on a really simple answer to, I think it was a question about whether you should do loops like "<" or "<=".  It was like a really simple little thing but he just felt like 46 up votes was a lot for sort of an ephemeral thought like that.

[3:52]

Spolsky: Was that a question that -- why did that question have so much attention on it?

Atwood: I don't know. 

Spolsky: Why were many people liking that question and voting it up?

Spolsky: [grunts encouragement]

Atwood: And Jon I thought had a very nice considered opinion about this.  And he's onto something because, it's like I said; until we have the reputation cap it was sort of a huge looming problem.  So I would say that's the number one thing I would look at.  That is you have to have really strong limits in the system. 

Spolsky: Maybe they should be logarithmic or something like Richter scales?

Atwood: Ya, I thought about that but it just it gets... hairy.  It just seems complex at some level versus the one man one vote rule.  I mean there's some people who think that the more reputation you have the more your votes should count which I think is extremely bad, like I think that's totally the wrong way to do the system.

Spolsky: Wait wait don't answer that yet because that's a question we have coming up.

Atwood: Oh really?

Spolsky: We'll come back to that.

Atwood: Yes, We'll come back to that.

1. Have some sort of built-in limits.
2. Try to make it so that other people can give you reputation.  It's like the whole Google model of PageRank where you building a website: you can't vote for yourself only other people can vote for you.  I think that's really important as well.

So I would say those would say those would be the two guidelines for anyone asking my opinion on these things.  Although I guess we have not seen the ramifications of the system that we have built yet.  But so far, so good.

Spolsky: I think it would be fair to say that it was definitely our goal that this thing - or it was our expectation - that these things would change all the time.  That we would constantly be changing things.  Adding new badges, removing badges, adding scoring rules. Right?

Atwood: But as you have noted before, once the system gets out there and people get used to the way it works there's a lot of bellyaching when you change things, there's some inertia that you start to get in the system around the way things are so it's difficult to just radically change course.  Our hope was that during the private beta we got all that stuff out of our system - all the major, crazy changes.  But we'll see.

Spolsky: Oh I got a cowbell.  [ding ding ding]

[6:54]

Atwood: Oh. [laughs]  Did the cowbell arrive?  Did the tambourine arrive?

Spolsky: Yeah, oh tambourine, I'll take that out for the next question but [ding ding ding] that's the cowbell.

Atwood: By the way, these aren't random.  These are actually Rockband accessories for the vocalist.  They're actually part of the song where you can hit the cowbell and actually have it register.

Spolsky: I was looking for the USB port and this just looks like a piece of metal.  I was not seeing... I thought this was going to be a USB cowbell.  I guess it's wireless or something?  I don't see where you put in the batteries.

Atwood: I'm excited that you are using it as a prop on our show though.  That makes it all worthwile.  It's excellent.

Spolsky: I've got all this gear here now, for those that don't pay any attention to my website, I wrote up a little article last Friday about what all the gear is here in the new studio setup.  I've actually got two mixers, and a recorder, and all kinds of USB... It's a lot of gear and the joke is that I think this entire thing could be done in software if I could just figure that stuff out.

Atwood: [laughs] Yeah, but it's fun to have knobs and dials.

Spolsky: The knobs and dials are fun and there are flashing lights here that don't mean anything.  For example, one of these mixers has a flashing light that says 152.2 and I don't know what that means.  I think that's the number of beats per minute coming from you, Jeff.

Atwood: But it looks cool.  It looks complicated and interesting.

Spolsky: It does. I personally believe that almost anything you try to come up with will over time tend to create people gaming the system a little bit and maybe non-optimal behavior that is "point optimal" but not "site optimal" like it optimizes for somebody's personal karma not for the actual outcome that you want.  And as that starts to happen you have to constantly adjust and tweak these systems, kind of endlessly.

Atwood: Right.

Spolsky: That's what happened with PageRank obviously and Google's ranking algorithm.  They pretty much had to keep continuously adding new twists. What does the knob do on the cowbell? [ding]

Atwood: But the central thing is still there.  I mean, I agree that there's been all these new--It's funny because blogs were kind of an exploit of the PageRank system in my opinion.  Blogs, just the way they are constructed, and the way people form these communities and link to each other, just naturally--they're not trying to game the system.  But it was a huge PageRank farm, effectively.  I know Google did several updates related to blogs because it got ridiculous.  I remember some bloggers had PageRank 7, which is astronomical.  It's a logarithmic scale right?  On one end you have apple.com and some really major sites like some of the Google sites are PageRank 10--Wikipedia, things like that, and then PageRank 1.  By the time you get to 4 and 5, you're a pretty serious website and I was just surprised to see bloggers at 7.  I love my fellow bloggers, but good Lord.  They're not institutions unto themselves, although some of them might think that they are.

Spolsky: Well, I for one have PageRank 7, so there!

Atwood: But you're not a blogger.

Spolsky: Oh yeah, that's right.  I'm sorry.

Atwood: You always tell people over and over again, so you don't count.  You're a writer!

Spolsky: I notice you only have 6 Jeff. [laughs]

Atwood: You're one of those fancy writers, with your words and your sentences and your paragraphs.  I just use bullet points and pictures.

Spolsky: Maybe I don't mind [not] being a blogger anymore.  I was thinking about the definition of a blog and how it's sort of changed a little bit.  And it used to be more about the fact that you were linking.  There used to be some implication that if you're a blogger that what you're doing is like, I don't know, Glen Reynolds, where everything you write is two sentences and it's just linking to other people writing longer form things.  There used to be a little bit of an implication of that in the idea of blogging, now there isn't so much.  I think now there's all kinds of different ways that blogs work, I think that's no longer--the word either has no meaning or it only means the reverse chronological order thing.  Hey, let's not talk about blogging, that's boring.  Let's talk about podcasting!

Atwood: Yeah, that's much more exciting. [laughs]

Spolsky: That's what people talk about on podcasts.  So anything else we want to tell Chris about karma systems?  I think you do have to adjust all the time, I think you have to come up with the first good guess and you just have to be ready to tweak it and watch what people are doing, what people are saying.  Part of the reason I even thought it was important to have the Stack Overflow podcast is that I really did want people--this is something the community will pay attention to.  They're going to say "Hey, I have an idea.  Maybe I should get 6 points when I do this, and only -2 points when I do that."  Having those conversations continuously is important for us to hear from the users of the site.

Atwood: It is.  It's important for them to go on User Voice and enter it, and then I'll decline it immediately.

Spolsky: Or just, even better, call into the podcast hotline.  By the way, I've mentioned the phone number a couple of times because I want to encourage more calls.  646-826-3879.  That's 646-826-3879, and that's the number where you can leave a voice mail for us to play on the show and we need more questions.  When you do call, please tell us your name and try to keep it to under 90 seconds.  All right.  Another question?

Atwood: Sure, let's do it.

Spolsky: This was sort of mentioned--we touched upon this.

[12:05]

Unknown Caller: Hey Jeff and Joel--downvoting.  Basically not all downvotes are created equal.  People who go around just downvoting everybody, their voice shouldn't really have as much weight as somebody who is very selective.  So, what do you think about a system where the voter's reputation is a multiple?  So people who have better reputation, their votes count more.  What do you think?  Bye.

Spolsky: So I guess you were very much against this idea?

Atwood: I think it's very, for lack of a better word, undemocratic.  I mean, the idea that your vote counts for more than mine?  There was a gem of something in there that's a little bit interesting that we've thought about which is, we have had incidents of people who--they get pissed off at another user so they go in and downvote everything that user has ever done.  Right?  First of all, that's not as bad as it sounds because an upvote is worth five times as much as a downvote, so it would take five downvotes to cancel out one upvote so there's a limited amount of damage you can do.  Plus you only have 30 votes in a 24 hour period anyway, so you can only do that on 30 posts by that user.  But still, it is aggravating and I totally empathize with people that are complaining because this is annoying.  Somebody told me, it might have been you--I don't remember, Mr. Reddit fan, but on Reddit they track this and if they see a user consistently downvoting another user, they basically silently undo all those votes, like those don't even count.

Spolsky: I do not remember saying that, but it just occurred to me that we should get--you know who we should get on as a guest?  Some of those Reddit guys like Steve or Alexis.

Atwood: Your bestest pals.

Spolsky: My best friends.  Alexis--He's in a band called Breadpig.  Are you familiar with that band?

Atwood: I am not familiar with Breadpig.

Spolsky: They play the Rockband instruments without the X-Box.

Atwood: Wow, that's cool.  That's very cool.  Yeah, definitely let's get him on.  That's doubly awesome now.  I'm totally convinced.  Joel and I have been talking more about getting more guests on because I think it just livens up the show and plus we get to broaden our horizons a little bit which is always kinda nice.

Spolsky: Yeah and we get better ratings when it's just not this old couple, you and me.

Atwood: [laughs] Whiling away the hours, sipping on our Bartles & James on the porch.

Spolsky: [laughs] That's what it's going to be eventually, isn't it?

Atwood: Yes.

Spolsky: We just need to find a place where we can both move.

Atwood: Yeah.  One final thing on that question.  Another thing that has been floated, and I guess you've kinda talked about this is just sort of a logarithmic thing.  Instead of something having a bajillion votes, where another post has four, some sort of normalization you do where your deviation's above the mean--and I'm kinda open to that but again I think it's kinda complicated to figure out versus the whole "one man, one vote" thing, which people kinda get.

Spolsky: What I'm wondering is, what's it going to be like when there are people out there with 100,000 karma?  Are new people going to come in and say "Well, I can never achieve that kind of karma, so there's just no point" and just lose interest?

Atwood: Another thing that's been floated, and I think some communities do this--once you get to a certain level, you just have "Max."  That's the number--you're at "Max" and then--we could cap it that way.  So it doesn't feel like oh--because someone will eventually have 20k rep. which is just huge.  We're almost to the point where someone has 10k.  I don't know, maybe we should have a cap?

Spolsky: How about nobody can have more than Joel? [laughs]

Atwood: [laughs] Because the intent of the system is that we trust you.  Reputation doesn't mean that you're a genius or that you know everything there is to know about anything at all actually.  It just means that Stack Overflow, the system, trusts you.  Once you get to a very high level of trust, you're just infinitely trusted.

Spolsky: I don't really like the "Max" idea.  I feel like you should always have something more to strive for.

Atwood: Yeah. No, I--

Spolsky: "Max" is like the President of the United States of America.

Atwood: [laughs]  Well, we've also thought about maybe extending some moderator priviledges to other users just so that if we're not around--the four moderators that we have--although to be fair, to be clear, almost everything in the system, users can do the essential things that need to happen on the site.  But there are some unusual things that might happen.

Spolsky: I've never really used my moderator priviledges, except for once and then I messed it up because I thought I was just flagging something and I was actually banning a user for life, [laughing] I think--I don't remember.  I'm not really moderating properly.

Atwood: Yes, that user felt very chastized by that, accidentally.  He actually emailed me, that was kinda funny.  What Joel is referring to is, normally you can vote something "offensive" or flag something "offensive," and once you get to a certain threshold of offensive, it's soft automatically deleted and so on, and so forth.  What Joel didn't realize is as a moderator, your offensive vote is binding and permanent.  If you cast an offensive vote, immediately you reach the threshold--one of the very few special things about moderators.  So when Joel did that, he banished the post.  He thought he was just casting an innocent little flag, but, no, he was removing the post.

Spolsky: Yeah.  I have a question.

Atwood: Are you in the list of questioners?  Did you call the number?  Did you leave your name?  Did you--

Spolsky: No.  All right, nevermind.  Sorry, I won't ask it.

Atwood: Yeah.

Spolsky: Let's see... I kinda got a new system here so I'm--repuation, downvoting--I feel like you should be able to go on forever just earning some kind of additional points but maybe not in a way that discourages new users from--maybe there's something about--maybe your karma kind of rots if you don't keep it up to date.

Atwood: That's another thing that's been floated, but I don't know.  Gosh.  Some of this stuff--I feel like you'd just have to throw it out there and try it and see what the reaction is.  But some of it is so broad and so sweeping, it's such a major change to behavior that--gosh, I worry about that--particularly that one.  If people come in--because already people freak out if their reputation changes by any way that they don't understand.  So if there's this global thing where every week everybody's rep drops, unless they participated--let's say you go a week without doing something so your rep would drop, well that's like--

Spolsky: Well, maybe you have two kinds of rep.  You know, you've got "old" rep, which is shown in Roman numerals and you always get to keep that.  [laughs] But it's old fashioned which is why it's in Roman numerals.

Atwood: [sarcastic] Yes.  That's great, that sounds simple and very easy to understand, the system that you're describing.  I don't think anyone would be confused.  I don't think I would get any emails about that.

Spolsky: [laughs] You'd just have to explain it in the blog.  Let's go on to another question, here--this is a good one.

[19:03]

Jeff Metzner: Hi Joel and Jeff, this is Jeff Metzner from Exton, Pennsylvania.  I'm a C++ programmer and here is my Stack Overflow related question.  It's about anybody being able to edit any post on the site, reputation dependend of course--which I think is a good thing, especially for the technical questions.  However, it just occurred to me that I recently answered a question about job experiences with a personal anecdote (whether such questions should be allowed or not is a whole separate question which we could come back to some other time) but it occurred to me that I answered this question with a personal anecdote and then if anyone is allowed to edit that, then that means that I could go back some day and find something with my name next to it that actually does not reflect my experience, which with a technical question is not such a big deal because presumably they would be correcting my mistakes.  But with a subjective answer they're potentially putting words in your mouth.  So, I was wondering what your opinion was about that.  Thank you.  Bye.

Atwood: I think that's a great point.  Joel, did you want to run with that?

Spolsky: Well, you've thought about this, so--no actually, one thing is that the personal anecdote--I kind of have to see the question--but what we've really been invisioning is the Wikipedia of the long tail of programming questions.  So a good answer does not necessarily--should not--this isn't a discussion group, it's not a place to tell your life story.  It's a place to provide encylopedia like answers to questions.  You know, the best answers are the ones that are the most--what's the technical term I'm looking for--the most encylopediac.  You know, what are the qualities of an encylopedia article?

• No point of view
• Balanced clear explanations
• No assumptions about what the reader knows or how advanced they are

Atwood: Have you been to Stack Overflow recently?  I mean, I'm just curious because what you're describing I've got to tell you--

Spolsky: Doesn't exist.

Atwood: No, some of them are like that.

Spolsky: I don't know.  The ones you hate the most are all the subjective ones, which are like "What's everybody's favorite keyboard?"

Atwood: Wait, wait, wait.  First of all, hate is a strong word.  I don't hate anything on Stack Overflow.  I may dislike and disagree with some of the things, I don't hate anything that's on Stack Overflow.

Spolsky: Your pet peeve is the subjective ones.

Atwood:  I hate you, Joel Spolsky, but the stuff on the site I stay more apart from.

Spolsky: [sniff, sniff] Sniff.

Atwood: You're very--he's very sensitive.

Spolsky: I'm putting your volume down 20%.  [laughing] Which knob is this for Jeff's volume?  Let's see... here we go.

Atwood: I think a lot of the tension in Stack Overflow--and I think it's an interesting tension--comes from the fact that we're trying to blend these two things that essentially are like oil and water.  One is the concept of shared ownership like Wikipedia right--where nothing is signed, right?  There's no article, it's like--this is the asphalt article by Joe Schmoe.  It's just, this is the asphalt article.  So you give up identity.  There's no reputation system.  There's certainly no upvoting on Wikipedia.  And then on the other hand, we have all these things in our system.  So we're trying to serve two masters, and I think it can get sketchy, right?  I mean, there's places around the boundaries of our system where it doesn't really fit very well.  But I think again, it depends--I guess it depends what you're putting in and what you want to get out of the system.  I mean, if you put in a lot of this really subjective stuff, then yeah.  But I think there's also the risk for anything you put on Stack Overflow--somebody could change it to "I'm an idiot," right?  You go in and write a really nice response and then I can go in and change it to "You know what, I'm an idiot."

Spolsky: Yeah.

Atwood: But it's all visible, it's all public.  And when users do bad things, what I love about Stack Overflow is there's nothing "moderator secret-y" that I really do.  There's a couple queries that we have that eventually I want to expose to people.  But I would just go in and look at a user's history just like you would.  If I see a user going in and changing a bunch of people's stuff to "I'm an idiot," the first thing I'm going to do is delete everything that user ever did, right?  So that user effectively disappears from the system and that's like a one-click operation for me.  Literally.

Spolsky: Wee, I like it.

Atwood: Yeah, I know, it's awesome.  So I can undo lots of vandalism really quickly.  And this is again, if I look at the--

Spolsky: Wait, sorry, Jeff you have a one-click way to delete all the changes--like somebody's gone and edited 30 things, you can rollback all their edits in one click?

Atwood: That user and everything they did ceases to exist.

Spolsky: Wow.

Atwood: This is obviously why that's something I'm really nervous about extending that to anyone outside our little kabal.

Spolsky: And when they get 148,000 karma...  You know what we could have badges for?  We could have badges for, like, "Real Address."  Like I have proven that you live at a certain address.

Atwood: We could.

Spolsky: And then certain things could only be turned on if we can actually go to your house.  Not that we will, but we could.

Atwood: We could.

Spolsky: And then you just use one of these--there are these online services that will mail a post card to an address with a code on it and they type it in correctly, then you register that and they at least control the mail that goes to that address.

Atwood: Well, one of the ideas Michael had, I think two podcasts ago when we were in New York City, was the idea of making up Stack Overflow t-shirts and many little swag things that we would actually send to people once they get to certain rep levels, just for fun.

Spolsky: Yeah, and they would have to send a photograph of them wearing it, with a code on it, and then they could get--

Atwood: Well, once we have their address and stuff, they have sort of semi-proven--so that's along the same lines of, like, if we are going to send people stuff we would have to actually know who they are and where they live to some degree.  I'm open to that.

Spolsky: Swag.  Call in if you have any suggestions for what kind of swag we should have from Stack Overflow.  646-826-3879, see I'm doing it like the talk radio host guys.

Atwood: Right.  I don't think I have a perfect answer for that question.  I think there's always going to be that tension in our system because on some level it can't really be resolved.  I'm really not seeing a lot of those kind of problems.  I mean, we talk about them, and this is what I like about--

Spolsky: Yeah, it's more theoretical problem than an actual problem.

Atwood: But even on my blog Coding Horror, anybody can post anything on Coding Horror, right?  I don't authenticate anything, and I have to go in manually and delete stuff.  It's just incredibly brutal 'cause my tooling sucks, it's a whole new story.  But honestly I don't have to do it that much, really.  It surprises me how little I have to do it, and you have a lot of people watching the system in Stack Overflow and if they see other people messing you up, they're going to be the good Samaritan and they're going to clean up.  It really does work.  I've seen questions come in that were just horrible questions--unbelievably bad.  And people go in and fix them, it's miraculous.  They take it, a bad question, they edit the title, they edit the body, they edit the tags and you end up with this really nice question with these great responses.  And it's all based on these little slices of effort that people are contributing, so I would hope those little slices of effort would also contribute to helping you protect yourself from malicious users.  Other people will help you in the system.

Spolsky: Mm-hmm.  Yeah, a lot of those cases, especially if it's just badly worded questions, bad English, bad grammer, written by a non-native speaker, etc.  And it's really cool that those get fixed.

Atwood: It's fun to fix other people's stuff.  It's really this collaborative thing.  One of my favorite things to do on Stack Overflow is find a user with reputation of 1 that has just asked a question that I like, and then I upvote it.  I get this little thrill.  "Wow, I just gave them their first badge."  I don't know, it's just crazy.

Spolsky: Hey, that reminds me.  One of the things that is underdone, under utilized, one of the features that is under utilized at Stack Overflow is just the voting on questions in general.  I think a lot of people don't even get why it's there, they don't vote, it's not--you know what I mean?  Voting for questions, not for answers.

Atwood: Right, no, I'm with you.  I think the more sophisticated Stack Overflow users do get voting for questions.  And there have been actually several really good topics on Stack Overflow.  They were meta-topics, but still they were good.  Like, why you would vote for questions and how you should vote for questions.  Of course, getting people to read that is another story entirely.

Spolsky: Well, I'm looking at the "Hot" list right now, the current "Hot" list, and you know there's some questions with like 13 votes, but I mean we really need to get, like, 100 votes for the good questions because I really want the "Hot" list to be--and maybe this is just because nobody comes in and the "Hot" page isn't the default, not enough people paying attention to the "Hot" list--I really think the "Hot" list should be the default, not the newest, but my feeling is that the "Hot" list is a place where you go and you find--

Atwood: The "Hot" list is the default.  It is the default, yeah it is.

Spolsky: Oh, it's just second.

Atwood: Oh, you want it to be the very first tab.  So you want it to be the first tab--

Spolsky: I just misinterpreted that it was not the default because it wasn't the first tab, and because it always remember what you stuck it on.  So I'm never going to see what the default is again in my life.  But my idea of the "Hot" list is that if you don't have any questions and you just want to learn something new about programming, that those would be things people voted up because they were just of general interest, you know?  I thought they were just interesting questions.  Somebody thought they were interesting questions, and that list would give you a whole bunch of interesting things to learn about.  Actually it is almost working, you know, there are pretty interesting questions on there right now.

Atwood: No totally.  And I encourage people, anyone listening, please vote for the questions that you like.  Again, I've talked about this on previous podcasts, but I continue to see it over and over in the system.  Some people are really good at asking questions and they should get upvoted for that, and you should pay attention to that.  Conversely, I've seen people complain that they didn't like the answers they got and a lot of the time, I don't want to blame the victim kind of thing here, but I gotta say--

Spolsky: You shouldn't have been wearing that dress.

[29:26]

Atwood: Asking a good question is an art.  If you get really bad answers that are totally not what you wanted, you've got to really look at your question.  I know this just from writing my blog.  If I write my blog entries a certain way, I get these really bizarre responses that are not at all about what I wrote about.

Atwood: And I consider that my fault.  I didn't do a good job of communicating what the blog entry was about, and it's hard.  I'll be the first to say it's very, very hard.  It's just like writing, it's difficult, but you've gotta concentrate on asking a clear, focused question.  The first time you don't get it right, either edit the question and make it better or just say, "You know what?  I'm going to ask a whole new question, link it to the second one, and then just ask again with a better formation to get better answers."

Atwood: Have you read Eric Raymond's How to Ask Questions?  Like this epic, 100,000 word--

[32:21]

{to be transcribed}

Atwood: Edit me!

Spolsky: Edit me!

[50:14]

Atwood: Let's do the trail-out.  You want to say the number again?

Spolsky: What's the trail out?  If you have any questions for the StackOverflow podcast, we'd love to hear them.  Why don't you give us a call?  The number is 646-826-3879.  Don't forget to mention your name and where you're calling from and try to keep it to under ninety seconds, or you can record an audio file in MP3 or Ogg Vorbis format and email that to podcast@stackoverflow.com.  We've got a transcript wiki which is getting a little bit behind so I'd like to encourage you, if you'd like to do a service for the hearing impaired it would be really nice if you could go to blog.stackoverflow.com and click on the wiki link there for every podcast episode and edit that with the transcript of what we're saying here.  And finally, I think I should have one more thing there, but I ran out of things to say.  So we'll see you next week!

Atwood: See you next week.

[51:11]

Outro, Advertising