@Blaise – thats a very good point. I think you can absolutely disagree with someone and still find the reporting to be credible. Perhaps I don't agree with your point of view, but if its presented in a transparent manner, fact-checked and with necessary disclosures, there's no reason to think its a poor quality article. That is an important distinction.

Also, your first point is something we are thinking about at NewsCred – some method of rating the raters!

I suppose their answer would be that it'll balance itself out in the end…

But what about controversial topics? Like, abortion or the Iraq war or something. With a service like Wikipedia, the aggregate result of a multitude of contributions is often a compromise, which is ideal. Is a compromise always appropriate for a credibility ranking? I mean, if there's a disagreement about an author's credibility, then sure, a compromise is a great solution, but if there's disagreement about the *beliefs* the author holds, is that something different than credibility?

I mean, can you disagree with someone and still consider them credible? There's a difference between someone being dishonest or having their facts wrong, and someone just coming to a different conclusion than you based on the same credible sources of information.