Rating systems. Any reforms?

[Olesniczanin]

Yes, I'm aware that many people have already submitted it but I think this issue is worth thinking twice.

Look at this image:


I've given it a + (and many of you have done the same) because it is good and aesthetic.

And now look at this image:


I've given it a + becauce it is perfect in every respect: technically, artistically, composition, idea and so on.

[no offence KarateGirl, you are great artist too and I only talk about that particular artwork]

My point is that we have to vote + or -, hot or not. That is, we have to rate good image and perfect image in the same way.

As a result, the highest rated picture is in fact the most popular one, and not the best one.
Britney is more popular than Mozart but whose music is better?

So that's why I think the rating system should be changed. Yes, I know you developers are bored with such requests but I must say that.

That's my conception:
votes from 0 to 10. The average without three the highest and three the lowest votes would be the note.

It that case, Pantera's problem could be solved: No picture would be in the top only because it is old.

Markus, you said:

[MorningStar]

Firstly, the current artworks on the top aren't there because they are old. They are there because they have many votes as they were made at a time when many people visited TheBroth (due to the BBC and the Digg feature). These artworks now have many votes and since traffic at TheBroth has been very low in recent months, it takes a long time for a new artwork to move up the ranks. Bayesian rating means that it's not the number of votes that makes the rating, but the rating's believability is based on the number of votes in relation to the average number of votes per artwork.

[Dolph]

Hi Olesniczanin,

for what it's worth, I also find it frustrating not to be able to say 'artwork A is good, but not as good as B'.

There might be a blog post about this whole thing soon.

Cheers,

Dolph

[MorningStar]

I think the problem that some people have (including Dolph and Oles) stems from their own understanding or interpretation of the rating buttons. It's a psychological and emotional problem and I appreciate it. I acknowledge that probably everyone (including myself) at one point in time wished for more graded rating to be able to give more detailed feedback.

However, we never said how people should rate, and we won't do so in the future either. There is no prescribed way to interpret + and - when it comes to rating artworks at TheBroth. Some people think that their vocation is to act in a balancing manner, eg. rate artworks that are "too high" minus, even if they're actually pretty good.

Others give pluses in a more "encouragement award" style, for example if the artist is new to TheBroth or known to be 10 years old (or both!) and has made a comparatively amazing artwork.

It is this variance in art appreciation by individuals that makes the bayesian rating and ranking work.

As an example: Let's say an artwork has 100 votes. In the fine rating system that uses 0-10, the final average may have been a score of 7.0.

In the current +/- rating system, the variance that people use when they rate artworks may have caused the final tally to be 70 pluses.

The end effect is the same: The artwork will approach the exact same final rank, at the same speed.

Let me reiterate: I do see value in having a 0-10 rating system but its impact is psychological: Some users may think they act more fairly when they vote. Granted.

On the other hand, many people are also happy that when an artwork is "pretty good", for whatever reason, they'll give it a plus because they have no problem whatsoever to give artworks of entirely different qualities the same shiny plus - yours truly belongs to that group of people.

Thanks to there being so many different ways to approaching rating and artwork appreciation, it eventually evens out.

We need people that act as conscientious balancers, as well-meaning encouragers, as pragmatic standard-upholders, as drunk or clumsy accidental wrong-voters... you name it.

But what comes out after we have many votes, thanks to the wonders of the bayesian equilibrium, is a pretty good idea of the preference of the community.

And just as Oles pointed out correctly: Ranking will always reflect a combined preference or "popularity" and it will never mean or imply a difference in regard to any particular quality - only a very fuzzy, averaged, general preference of some artworks over others - and this fact cannot be changed, no matter how much more fine-graded we'd make the rating of individual artworks.

Personally, I feel as a compromise, the quicker "hot or not" solution wins with its obvious benefits over the 0-10 solution, which also has benefits, as discussed, but also some downsides. Yet again, which method individuals prefer as a method of rating will again vary quite a bit.

I feel that we'll need to agree to disagree here. In the future, it is thinkable that the way rating is done could be changed via a profile setting - that way we'd acknowledge that there ARE different ways of rating and that some folks prefer one over the other. After all, none of these methods is "better" than the other on an absolute level! We can do that technically because mathematically, the resulting rating and ranking is identical.

[GO GO]

well....If both of these voting methods(i.e +/- and 1 to 10 rating system)are basically same and produce same results, why not go with the one which gives more psychological satisfaction.

[MorningStar]

Well - which one does give more psychological satisfaction? And are we certain that all of us prefer psychological satisfaction over ease-of-use?

[GO GO]

in my case i would like the 1-10 styled rating
and i also think that it wont be that difficult to use either after all u have to see and click thats it....

[Olesniczanin]

I think it would make people see the full image before voting. it really annoys me when people are voting after seing a thumbnail only. I know it is more comfortable but...

[GO GO]

ya oles i agree i have risen this issue of thumbnail voting many times and as new user enter they kinda want many award pts and go on voting spree which may be unfair

[MorningStar]

I hear you. If we had 0-10 rating then it wouldn't be possible anymore to rate on the thumbnails, as there's just not enough room.

Ok - here's the deal. We'll soon create a survey and we'll add that question too. Speaking of survey - why don't you submit your own questions in order to address some of the issues that always bugged you? If the majority likes one thing over another - we'll certainly listen!

So for example, this question could be:

"For the snapshots, do you prefer a rating system from 0-10 or do you prefer the current rating system using +/- buttons?

(o) 0-10 system
(x) +/- system"

And so on. Actually, as I'm considering this, I wonder... we could even do that without great fanfare, directly in a dedicated forum. I think I shall create such a forum... called "User survey", and everyone can add their own questions and polls!

So, real question: What do you prefer, a special custom page on TheBroth or simple a forum dedicated to user survey questions and polls where everyone can answer what they want and add their own polls and questions?

(Somehow I think I can predict the answer to that question....)

[Olesniczanin]

I think there should be a forum topic where we could give all suggestions, imply questions and give arguments. After the end of this discussion you could create a special custom page with the survey (on the basis of our discussion).

I think that voting on a forum would be a bit messy and hard to sum up and add up. What's more, in my opinion it would be good to keep results of the survey in secret untill the survey come to end.

In addition, I think that more people would vote if there was a special page than on the forum. Many people don't visit forum at all.

[MorningStar]

Well, what can I say, Olesniczanin - I hear you.. Your suggestions are (and always were) superb.

We'll try and implement something that is as close to what you suggested as possible! Thanks for the input (I wish I could dish out maxi-pluses in the forum!)