NewsTrust, where people rate the quality of news stories, has launched a beta site. The potential of this approach is terrific: community involvement in understanding how well journalists — including bloggers — do at their jobs (whether it’s a professional or amateur activity or something in between).
Clearly, this is an early iteration. But the promise is clear.
Join up and help out.
(Disclosure: I’m an advisor to NewsTrust.)
on Dec 1st, 2006 at 5:48 pm
Dan,
Interesting project! thing I don’t like about it: the rating process is too much of a “black box” when it need not be so: why not have a “talk page” ala Wikipedia, for instance? (or the comments function for Digg) and let people talk it out… ask each other questions, point out flaws that others might have missed… and make all this viewable not only by those who want to rate (which would very likely result in more accurate ratings) but also by those who just want to read the stories.
Of course, an overall rating number is very useful if you want be able to quickly pick a couple of stories worth reading — it’s just that in plenty of cases the rating number may be meaningless without an insight into what were the raters thinking…
Also, some sort of guidance as to what would constitute 1, 2, 3, 4 and 5 in the rating categories given would probably help a lot. Maybe they could gives examples (5 stories in each rating category that, according to whoever runs the site, would rank 1, 2, 3, 4 or 5 — it would probably help to ask users for input with this). I think some sort of a common ground as far as rating steps (1, 2, 3, 4, 5) needs to be established or most people are just going to rate “safe” and pretty much all stories would end-up being rated “average,” which would be useless…
Delia
on Dec 1st, 2006 at 7:46 pm
Delia makes some good, thoughtful points. Much appreciated.
We’d love to turn on discussions for every story, but not until we get some funding. We’d have to hire a full time person to monitor hundreds of new discussions per day, and can’t afford this quite yet. But you can post your comments on any story and it will appear next to the other comments for that story. And as soon as we get funding, this will be a high priority for us.
We’re working on providing more guidance on how to rate stories. Note that if you click on any rating question title (eg: Fairness) in our review forms, we give you specific tips on how to rate that particular quality.
So far the concern about too many people voting 3 for neutral has not been an issue. Most of our reviewers seem know to hit 4 when they think it’s good, or 2 when it’s bad. In fact, we have the opposite problem, with too many people hitting 5 when they really like a story. In time me may calibrate everyone’s ratings to adjust for their personal rating style.
Thanks for your insights and constructive feedback. Keep’em coming!
You can also share your thoughts with other NewsTrust reviewers on our suggestions page:
http://beta.newstrust.net/about/site_feedback
Muchas gracias.
on Dec 2nd, 2006 at 10:35 am
Hi Fabrice! You are welcome! why moderate the discussion? (especially if you don’t have the funds for it) — just make the function available AND make it clear it’s there so people can *discuss* among each other… not just post comments (that’s good too but not as useful as having real discussions).
The tips on rating are good (good categories!)– I just think they are a bit too general (the 1, 2, 3, 4, 5 levels for each category are just left undefined) to be very useful to the rater or to those just reading the stories and trying to make sense of the rating.
re: “know to hit 4 when they think it’s good, or 2 when it’s bad”
well… you want them to do more than just give you their gut reaction… (I *think*… — I mean, *I* would… I think it would be much more useful…)
Oh… and, I think it would be good to also have a category by category breakdown of the general score available (maybe a “breakdown” link next to the general score?). Some people may not want to read a story that rates poor at the factual support level but somehow manages to do OK overall…
Further along this idea… I’d consider letting the reader pick the criteria: maybe have a search box that lets the user get a list of stories according to their desired criteria (e.g. no less than 4 on factual support, don’t care about originality etc.)
re: hitting 5 when they really like a story
I’d get rid of that question altogether (re: “how good is this story?”) or at a minimum I’d have it at the end — not at the beginning — (if you *must*…). I think letting the aggregate score on the individual categories give the overall “recommendation” would really help the rater with not allowing things like personal likes and dislikes get in the way (you want them to be as objective as possible… I *think*…). In fact, a message asking them to “please don’t let your personal like or dislike of the story affect you rating (if you don’t think you can do this, please skip this story)” would probably help.
re: “In time me may calibrate everyone’s ratings to adjust for their personal rating style”
I think that would be much more difficult than establishing a common understanding for the rating steps in each category (1, 2, 3, 4, 5), but, of course, I could be wrong…
Delia
P.S. I’ll try to keep an eye on your project but I’m trying to not get sucked-in any particular project. D.
on Dec 11th, 2006 at 9:36 am
Hi Delia,
Experience has shown that if you are going to have thousands of open discussions on the web, you need a minimum amount of moderation to make sure these discussions don’t get filled with spam, abusive comments or flame wars. Note that the very blog we’re having this discussion on is moderated, for very good reasons. With the volume of new stories submitted each day to NewsTrust, we would add thousands of new discussions per month, each requiring some monitoring. That’s at least a part-time position, turning into full-time in a matter of months. For now, topic discussions will make this a lot more manageable.
We will improve our story review forms to offer more feedback on how to rate each criteria. We already display a breakdown of ratings by category on each story overview page. We are considering letting members personalize their own story listings according to different criteria, in coming months.
The reason we keep the first recommendation rating question (“how good is this story?”) is to measure its overall popularity, so we can calibrate our measurements against other services who only track popularity, not quality. And we think popularity counts for something in determining overall quality, even though it’s only one of over a dozen rating criteria.
Thanks again for your good feedback. If you have other comments about our service in the future, I suggest you enter them directly on our site, so we can respond more quickly, and let our community participate in our discussion:
http://beta.newstrust.net/about/site_feedback
All the best,
Fabrice