Center for Contributory Science

This article describes the Center for Contributory Science (CFCS), an imaginary journal, which I envision as the next generation scientific literature. See my previous post for the motivation for a next generation scientific literature.



The CFCS submission process

Submitting a paper

Scientists are encouraged to submit rigorous scientific research for publication in the journal. Choose the subject category appropriate for your paper. Based on your chosen categories, an editor with appropriate expertise will be randomly assigned to your paper.

Authors must first review / contribute

All authors must be registered CFCS users. Before manuscript submission is finalized every author on your manuscript (including the corresponding author) must do one of the following:

• If there are any papers in Limbo in a category the author feels qualified in, the author must review a paper. Authors on the same manuscript submission can't review the same paper. (Papers are presented oldest to newest to prevent any one paper from remaining in Limbo for too long. See the section below, "the status of a manuscript", for details on Limbo)
• If an author claims they are not qualified for any of the papers in their qualified category, the papers are placed in the author's public "not qualified for" list along with an optional comment by the author (this public acknowledgment prevents people from always claiming they aren't qualified to review papers).
• If the author does not feel qualified to review any of the available manuscripts in Limbo, the author must score any 3 papers in Purgatory or Heaven with a thumbs up or down and a corresponding comment to each score (see below for definitions of Purgatory and Heaven).

The above features of CFCS aim to ensure

1. there are at least as many reviewers as there are papers (and most likely many more)
2. authors along for the ride at least have to contribute to the review process
3. professors can't get out of reviews (and get credit for reviews) by sending work to their students
   1. students get credit for their review work getting their name out early
4. if you want to submit 100 papers in a year, you must be willing to review 100 as well

For details about how the review process works at CFCS, please see the section "The CFCS Reviewer Process" below.

Authors decide a direction for their manuscript

Upon completion of the review/contribution requirement by all of the manuscript's authors, the manuscript submission will be finalized. Authors may then send their submitted manuscript on Purgatory track or on Heaven track (see below for details on Purgatory and Heaven).

Editors decide paper status

Editors decide if a Purgatory track paper goes to Purgatory or if a Heaven track paper goes to Limbo. This editorial step is simply to weed out complete rubbish before it goes to review. Almost every manuscript should pass this minor screening.



The CFCS reviewer process

Reviewing a paper for CFCS works in a similar manner to most contemporary journals. However, reviews are not anonymous and are publicly visible with the manuscript upon submission. Reviewers do not have to be authors. Any user can do a review to get a credit, so they can later submit a manuscript without having to review. Ideally in the CFCS system, few if any reviewers must be asked to review a manuscript by the editor.

In general, reviewers choose the manuscripts they want to review from the set of all manuscripts in Limbo they feel qualified to review. Each manuscript in Limbo requires four separate reviews. Upon receiving the authors' revised manuscript and response to the reviewer comments, each reviewer places a vote to decide if a manuscript belongs in Heaven or Purgatory. The reviewed manuscript goes to Heaven if the manuscript gets at least three out of four reviewers suggesting the manuscript for Heaven. In the case of a tie, the editor holds the tie-breaking vote, which he casts upon reading all four reviews.

All reviews and the authors' responses to the reviews are publicly available with alongside the final manuscript. Both the original and the revised manuscript drafts are available as well.



The CFCS editor process

Editors are either

1. reviewers whose quality reviews have gained them a large reviewer impact score and who agree to the job
2. invited editors (if there aren't enough high ranked reviewers)

Editors decide if a Purgatory track paper goes to Purgatory or is sent to Earth. Editors decide if a Heaven track paper goes to Limbo or is sent to Earth. The main job of the editor is to eliminate rubbish (pseudoscience and just bad science). Editors must also decide if the subject categories selected by the authors are appropriate. Most importantly, editors hold the tie-breaking vote when there are two Heaven votes and two Purgatory votes from the four reviewers. In cases with no tie, the reviewers alone decide the final destination of the manuscript.

The CFCS user process

Any registered user can score and comment any papers, comments, and reviews besides their own. A reader cannot score a paper, comment, or review without leaving a comment to explain their score. Scores and comments are publicly available with the manuscript and on the users' CFCS page.

All users have a reviewer impact score, a comment impact score, and an author impact score.



The status of a manuscript

The status of a paper follows two of the key ideas of CFCS: 1) information is always public; and 2) information is never deleted. Everything that happens to a paper on its route to Heaven is recorded and posted for all to see. All reviewer comments, all responses to reviewer comments, and both versions of the manuscript are available for download.

A publication search in CFCS can be limited to certain types of papers (to allow for example only peer-reviewed work) or it can draw from all of the CFCS library.

Heaven

Heaven is the pinnacle of CFCS. Manuscripts in Heaven have been peer-reviewed by four reviewers, the authors have responded to the reviewer comments to improve their manuscript, and the manuscript received a majority Heaven vote from the reviewers. Voting is carried out by the four reviewers plus the editor. The votes do not become public (to the reviewers or the editors) until all the votes are in (to prevent biased voting). Papers in Heaven are charged a modest processing fee to allow them to be uniformly typeset in the style of the journal. Typeset papers are submitted to pubmed. The four reviewers set the initial paper impact score with their votes. These initial seed scores count double the normal reader submitted score. Once entering Heaven, the manuscript can be scored and commented by all readers of CFCS to adjust each manuscript's impact score.

Limbo

Manuscripts for peer-review work are initially sent to Limbo. A manuscript remains in Limbo until it has received the necessary number of reviews, responded to those reviews, and been voted into Heaven. Failure to respond to the reviewers (within a fixed time) and failure to receive a majority vote result in the manuscript being sent to Purgatory.

Purgatory

Purgatory track manuscripts only need to pass the editor's inspection (otherwise they go to Earth). Purgatory is an option for works where the authors don't want to go through peer-review. Examples of good pieces for purgatory include: reviews and reports of failed experiments. Upon entering Purgatory, the manuscript can be scored by all CFCS readers to determine its paper impact score.

Earth

Manuscripts not passing the editor's initial quality screen go to Earth. Authors get one petition to get out of Earth and back into Limbo or Purgatory.

Hell

Manuscripts discovered to be fraudulent go to hell. (perhaps papers where the equation to word ratio is greater than one belong here too?)


Definitions

• score: a vote by a CFCS reader; a score can be positive (thumbs-up) or negative (thumbs-down); reviewer comments, reader comments, and manuscripts can all be scored by all CFCS readers; all scores must be accompanied by a comment where the reader explains their reasoning for the score
• review: similar to the current scientific literature, a review in CFCS aims to strengthen the quality, rigor, and focus of the submitted manuscript; reviews are publicly viewable with the manuscript as are the author's response to the review
• comment: a comment is a CFCS reader's written opinion of a manuscript, review, or another person's comment
• reviewer impact score: for each individual, this metric is determined by the number positive scores minus the number of negative scores from CFCS users for all of the reviews written by the individual
• comment impact score: for each individual, this metric is determined by the number positive scores minus the number of negative scores from CFCS users for all of the comments written by the individual
• paper impact score: for each manuscript, this metric is determined by the number positive scores minus the number of negative scores from CFCS users for that particular manuscript