Solving the Commenting Problem for Science Articles

There's been an inordinate amount of hand-wringing in the open science community over the commenting of science articles, or the lack there-of. Whilst great post-publication reviews have occurred on science blogs, scientists in general don't comment on articles. It's such a problem, we've had surveys and even Wikipedia cries about it.

Why is that?

Some think it's an institutional problem, others think it's a technology problem, I now think it's a UI problem, and I believe that I've finally come up with a workable solution.

Why has commenting failed so far?

The irony is that whilst scientists push the envelope of knowledge, they're anachronistic in the use of technology not to do with their speciality. Most people in the scientific community are slave-dogs, not playful first-adopters. When I was working at UCSF, many people were surprised I even had a blog. Image if everybody spent their time blogging, no real science would get done!

What this means, is that scientists are no different from *gasp* other people. The problem of getting scientists to comment on scientific articles is the same as any web startup trying to get people to do something new. Whatever system that will succeed has to satisfy both a pressing need and do it in a convenient way.

Alas, the commenting systems I've seen fails in either of these two ways. Either it satisfies a need but is inconvenient to use, or it is convenient to use but does not satisfy a need.

Let me explain. I can only see two possible needs for a commenting system: # Personal notes on an article for future reference; or # A live discussion with a bunch of insanely interested people.

Now consider the commenting system in the PLoS journals, the commenting does not fit into anyone's natural workflow for the simple reason that hardly anyone I know reads their articles on the webpage. Most people I know either read their papers in a PDF reader or print it out on paper. Once you look at it in that context, it seems ludicrous that someone reading a paper will jump out of their PDF reader, go back to the webpage, wade through the registration, find the commenting input, then comment on what they think.

For comments to be convenient, it must be built in to a PDF reader such as in Papers or Mendeley. You can bet your bottom dollar that they are working on this right now. It will probably end up looking like the highlighting feature on the Amazon kindle.

Annotatr: my attempt at a commenting website

But then there is the other use case: commenting as discussion. For that you need a large user base. As most journal websites don't have many users, the difficulty is to bring all the users together. However, one place where there is a huge number of users that cuts across a lot of journals is the online citation manager CiteULike.

Why wouldn't CiteULike just throw down a commenting system on the website? They actually do, but it's buried amidst all their controls and not particularly user-friendly. Still, it's 2011 and the problem of designing good commenting system is a solved problem. Whilst I believe that the Reddit commenting system is best of breed, the third party plugin Disqus is almost as good.

So I built a mashup annotatr, which glues together CiteULike and Disqus. It was a short fun project involving scary technologies such as AJAX and JAVASCRIPT, and hosted on the GOOGLE APP ENGINE. Annotatr provided a simple interface to find abstracts to science articles and then to comment on them. I even got my buddy Mark Reid to add some design sizzle to the website. Whilst, there was a moderate amount of buzz for it when I first created it, after a week or so, I got ... a big fat silence.

I was afraid that I had built yet another dead project on the internet.

The missing component, time-criticality

A couple of months later, I saw some activity flare up on the Annotatr. Mark had taken the annotator code and morphed it into, which was a commenting system for the annual Machine Learning conference. It worked great, and I got some flow-on traffic from there.

However, Mark's site, a few months after the conference, also fell dormant.

Then I finally realized my mistake, I had built Annotatr as a note-taking site, which does not fit into anyone's workflow. What I should be doing was retooling the website as a place for discussion. If you don't have a group to discuss, you won't be motivated to discuss and that's what conferences are great for.

The challenge was thus: how could I recreate online the feeling of a conference?

You need a bunch of like-minded souls

Then I read this article on Joel Spolsky's StackExchange, the new open Q&A site that has taken the programming world by storm. One of the things he discussed was that for a successful Q&A site to develop, you need a strong nucleus of people seeding the site. People (me included) are herd animals and generally don't like going where no-one else is going. Joel's solution was to create Area 51, a staging site for StackExchange, where passionate individuals would boot-strap a mini-community of people to seed a new Q&A site.

This insight also explains why you can get great commenting on blog posts, but not on journal websites. Great bloggers foster great communities in their readers, and the posting of a blog-post provides a time-critical phase for readers to react to each other, resulting in a real-time discussion.

If you want a discussion, you need to bring people together at the same time. I realized that I needed to start an online journal club.

Now let me digress a bit. When I started research, I used to love going to journal clubs, but much later, I grew to hate them. When I was a grad student, I had no real accomplishments, and the only place where I could flex my science muscles was at journal club. I would always try to sludge through the swamp of jargon, just so that I could ask the trickiest question I could think of at the end of the presentation. But then I realized that 1) I was being a dick and 2) journal clubs were quite boring. Either the articles were safe technical papers or people would not discuss the crazy fun issues. Often journal clubs became a boss-impressing exercise.

The problem was that most people in science aren't really into fun discussions of papers. If I was to create an online journal club using Annotatr, I would have to go and find the crazies.

How to collect people

To cut to the chase, I ended up finding a great bunch of people and started an online journal club, focused on Molecular Dynamics simulations. We use Annotatr to have discussions. I have now done it for 4 rounds (round one, two, three and four). I get an average of about 19 comments per round.

I love the diversity of the group, the 8 of us are involved in molecular-dynamics simulations, bioinformatics, chemoinformatics, NMR spectroscopy, membranes and cloud-computing. To seed the original group, I used the oldest social network technology in the book, I sent out emails:

Hey guys,

Here's an idea I've been playing around with.

I want to start a specialized MD journal club, using my article
commenting website

If you commit to it then, once every fortnight or so, you

  1. get a paper to read in 4 days.
  2. adding 1 or more comment on the website

The comment can be anything from, "kinda boring" to a treatise on their method.

Are you guys in?


ps. anybody else you can think of?

I picked people who had posted on Friendfeed or Twitter, and people I had met in conferences. Alas most scientists I know are actually publicity shy. Those who aren't, are already posting on Twitter or Friendfeed or Facebook. It turned out that only the ones who had previously been active on other public forums ended up contributing to the online journal club. So if you were to invite people strategically, stick to the loud and raucous ones on other social networks.

Time-constraints are important.

I find that an important part of running an online journal group is the use of time-constraints. Constraints coordinate people to comment at the same time. Also people respond well to deadlines, and short deadlines helps to maintain a sense of proportion.

After tweaking it a bit, I now space each round for every 2 to 3 weeks. Too often, it gets rather stale. Each round, I post a paper for everyone to read. Given the variety of work schedules, I ask people to spend 3 days to read it, then 3 days to comment. Everyone is expected to make one comment, it could be something as simple as, 'this is boring'. I've found that a 3 day period provides a relaxing space to think about a paper. After a few rounds, I was happy to see that people in the group were willing to pitch in, throwing recommendations my way.

Starting such an online journal club is now incredibly easy. I've written a list of steps to start one.

Now go forth and start your very own online journal club!