Review Process

The review process takes approximately two weeks and it is divided in 4 parts:
  1. Upon submission, the general chair assigns a reviewer from the reviewer pool — candidates are filtered based on the operating system they use and ranked depending on their current and previous load.
  2. The reviewer downloads the code and follows the instructions to replicate one of the results shown in the paper. If necessary, the reviewer may recommend changes to the repository and collaborate with authors to ensure that the guidelines are followed, communicating via email when possible or over the phone if necessary.
  3. The reviewer notifies the chair when a decision is taken.
  4. If the stamp is approved, the general chair notifies the contact author, the respective journal, and adds the paper to the list on http://www.replicabilitystamp.org.

Appeals

In the case where there are concerns with any particular submission either the General Chair or the corresponding Associate Chairs can be contacted for clarifications and help.

Conflict of Interest

In case the General Chair is conflicted with the submission (following the ACM SIGGRAPH guidelines), the review is handled by the previous General Chair.

Membership

The volunteer base of reviewers will be selected by the general chair and the Associate Chairs and will be available at http://www.replicabilitystamp.org. There will be one Associate Chair for each journal joining the initiative. Each Associate Chair will collaborate with the General Chair to ensure that the guidelines conform to each journal and conference expectations. Additionally, an Advisory Board will consult with the General and Associate Chairs on refining policies. The Advisory Board and the Chairs will meet yearly during the SIGGRAPH conference. The members of the advisory board will be added or removed with a majority vote of both the chairs and advisory board. The role of general chair can be held for not more than 3 consecutive years. An election from the chairs and the advisory board will replace the general chair. The role of associate chair is tied to the role of Editor-in-Chief of the corresponding journal. General and Associate chairs are automatically added to the advisory board when their term expires.

Review Guidelines

The replicability stamp initiative expects reviewers to:
  1. Make known to the requesting editor any possible conflicts of interest
  2. Review the submission by the agreed-upon deadline
  3. Understand the charter and reviewing standards and procedures of the replicability initiative.
  4. Maintain the confidentiality of the existence and status of submissions of which the reviewer becomes aware
  5. Not use results from submitted works in their works, research or grant proposals, unless and until that material appears in other publicly available formats
  6. Not distribute a submission to anyone unless approved by the editor handling the submission
  7. Maintain the anonymity of the other reviewers, should they become known to that reviewer.
Note that these guidelines are based on the ACM’s reviewer guidelines.

Back