Bayesian Rating - how to implement a weighted rating system

has an upper limit been established yet? i’ve noticed that many excellent works by many people aren’t moving up at all because of the fact that other good but not excellent works have so many more votes because they’ve been on the site for so long.

yet at the same time, its still possible to catapult collaborations to top 20 just by having everyone who worked on it (say liek 10 people) all vote it +.

Now this seems a very interesting use of Bayesian Theorem. I like the idea. I don’t know if its the most fair one but it certainly does not seem unfair to me..

Albeit it depends on being well-voted(voted by a lot of people) which is not always guarenteed, at this moment i cannot think of anything better.

Nice job.

Very good explaination of weighted averages. But wonder about all the queries that would take place for a large number of object. For example i want to apply this logic to rate songs. But potentially there could be many thousands of songs. So every time a song is loaded, you’d run this calculation again monay thousands of other songs (everytime). What if votes where collected per song and then a cron could talley up the days events - run/reset the averages once or possible twice a day. You’d reduce the server load and queries by a large factor I image. Of course you’d give up instantaneous results. What is your take on this. Thanks

Hi Joe, you’re quite right. Every single new rating, adding or deleting an item requires all rankings to be recalculated. For this reason, TheBroth just sets a flag if the ranking and rating values have to be recalculated, and recurring cronjob does the rating calculation every N seconds (if the “must_update” flag is set).

For other installations of Bayesian rating: Depending on the server load, capacity and amount of items to be ranked, this cronjob interval will require to be set accordingly to provide a good compromise between processing cost and data freshness.

Markus, thank you so much for this great article. I have been struggling with a very similar issue and was told me Bayesian stats might help. I did a search and ended up here. Exactly what I needed!

Well Thanks, such that atleast someone think about to implemention of weight rating system.

So Well you also give tutorial in the weight rating system. I need to clarify some queries about it

I’ve implemented your Bayesian weighted rating system on a website that I have built and it works like a charm - http://www.yolo.sg
Thanks for this article it helps heaps!

Ok, so we’re producing a Maximum Likelihood estimate of the parameters of a Gaussian distribution, with a conjugate prior.

Now, one can argue about priors until they invade your galaxy, but I would suggest that using avg_num_votes is a bit harsh. The actual weight given should probably be less. There are ways to estimate the correct weight… if you do this, what you are doing is Empirical Bayesian estimation. It’s a little more involved than what you’ve described here, but worthwhile.

Another neat thing a Bayesian approach gives you is your degree of uncertainty about a rating.

I think your SQL for item votes could be simplified a bit. The average of votes is their sum divided by the count. This is then multiplied by the count.

AVG(vote) * COUNT(item)

is really

SUM(vote) / COUNT(item) * COUNT(item)

which reduces into

SUM(vote)

You could turn that into a variable called this_sum_votes, and put that in place of this_num_votes * this_rating.

Patrick,

1. Yes, absolutely.
2. Yes that is the case.

In my algorithm, if the avg_num_votes were 100 and there was one item with 100 votes of rating 1.0, then BR = 0.75. This affects all items equally though, and since the goal is not to get absolute rating for each item but rather relative ranking, the actual BR value doesn’t matter, only the relative BRs to each other so you can rank them.

As many folks have pointed out in comments here and also on reddit, it depends on the system you’re working with. If you have a system that has a steady influx of new items, yet the average number of votes doesn’t change significantly, then you can use avg_num_votes as the magic weight.

And as I’ve written above, for other systems it’s preferable to use a constant or provide an upper limit. In comments on reddit I’ve seen other good suggestions, such as using a function that takes into account, in simple terms, that “N x ‘certain enough’ ” is still pretty much certain enough, and so you could use magic values such as the squareroot of avg_num_votes etc.

Hi Marcus, do you feel an error margin would be appropriate in a ratings system? If so, how would you suggest it be used?

According to the Wikipedia entry on error margins there are a couple of ways of calculating it for polls, but I’m not certain how or whether these would apply to a rating system.

[…] Bayesian Rating - how to implement a weighted rating system - Developer Blog @ TheBroth.com […]

[…] Weighting a product’s ratings based on the number of ratings received.  When a product has a small number of ratings, these ratings should count less than those for a product rated many times.  To achieve this, you can add a “magic value” into the algorithm that calculates a product’s average ratings.  This “magic value” brings products with few ratings closer to the average ratings of all products, then reduces its effect as more ratings are received, allowing established products’ ratings to float freely and more closely reflect the average of its ratings. […]

[…]  I’d like to be very clear about this right off the bat: I will not remove or reject overly negative or positive ratings on anything. While the site is effectively a dictatorship (well, from a management and editorial perspective, at least), people have a right to their opinions, even if they’re extreme. So, there’s definitely no point in asking me to remove ratings. No one has as of yet (though I have received a couple of complaints about what’s been happening, as well as a ream of comments on my profile), but I want to make sure I’m heard loud and clear on that front. I don’t think that’s a real solution to the problem, anyhow. While I’m pleasantly surprised to see people watching the author rankings this closely, it worries me that people are getting so hung up on their ratings. As I’ve reiterated to various people in the last day, the more ratings we have in the system, the less these negative or positive ratings spikes will have an effect. We’re seeing a major impact from it on author rankings right now because several of our top-ranked authors don’t have a huge amount of ratings, so the influence of 5 overly negative or positive votes on their branches or chapters will be much more noticeable. What I want people to keep in mind is that these abnormalities usually iron themselves out over time; as frustrating as they may be in the short term, they will slowly be less and less important the more ratings people add to the system. Now, as many people may already be aware of, 5-star ratings systems such as the one we have on Protagonize are inherently flawed in a variety of ways. At the same time, they are a simple, clean rating mechanic, and they comprise what is likely the most common rating system on the web in the last decade. The flaws with 5-star systems have been discussed to death, but if you’d like an overview, Life With Alacrity is an excellent blog that discusses ratings systems in depth. There are several blog posts there that are worth a read on the subject: Using 5-Star Rating Systems, Collective Choice: Rating Systems and Collective Choice: Experimenting with Ratings. What it comes down to is that 5-star rating systems are in many cases the least of all evils; if you can define the rating values clearly, make it easy to rate content, and accrue a large sample of votes, they become more valuable. However, as the sample size decreases, and the clarity of the different rating values becomes less obvious, votes tend to bunch up at the higher end of the scale and aggregate ratings become less valuable. I’ve also seen several people request that authors no longer be able to rate their own branches and chapters. I tend to disagree with this in principle, but I think that the two solutions I offer below may soften the impact of someone going through and rating all of their own posts 5/5 stars, without removing the ability to rate your own content altogether. Note that many large, well-known existing systems allow you to rate your own material, from Digg, to Reddit, to Amazon. I have a variety of different ways at my disposal to attack this problem, the first being implementing a Bayesian weighted rating system that would work hand-in-hand with the existing ratings. What this essentially means is that items and authors with less ratings would be weighted back towards the average, and statistical anomalies like a batch of 1/5 votes would affect users less. Of course, at the same time, a batch of 5/5 votes would also affect the overall rating less. You see the point. When I make this change, you won’t see anything different on the front-end, but you may notice that average ratings become a little smoother overall. The other option available to me is to move away from a 5-star rating system entirely, and switch over to more of an item popularity system. There are several ways I could do this, but I’m leaning towards more of a Reddit-style system of +/neutral/- votes. There would likely no longer be an average rating on stories, but instead an aggregate count of votes. This would remove a bit of the ambiguity from the current system, and ratings would be a bit more meaningful overall. However, every system has its pitfalls. If we go this route, it may be necessary to institute some kind of karma system to force users to be a little more conscientious when voting. What this means is that you’d be given a certain number of karma points based on the length of your membership, updated regularly, and you’d have to spend those points wisely to rate content. I’d like to hear what our members have to say about this matter; feel free to make yourself heard here and I’ll use this as an informal poll of the community on the subject.  Posted in Collaborative Writing, Features, General, Site mechanics […]

@Maxwell I think he meant SELECT item, AVG(vote) […]