Submission Requirements

To ensure a fair and efficient review of the code, the following guidelines must be followed when you prepare your submission. If there are technical problems that prevent you from complying with the requirements, please contact the Chair of the Replicability Committee.

1. Distribution

The software must compile and run on at least one of the following operating systems:
  • Windows 10 or newer
  • Ubuntu 20.04.1 LTS or newer
  • MacOSX 11 or newer
Note that you can pick whichever operating system you developed your code on. Your code does not need to compile on all of them to obtain the stamp. The code must compile and run on a vanilla installation, without any software installed. If you need specific software to be installed, you should provide a script that automatically downloads and installs all the required dependencies. In the case of Windows, a text file with precise installation instructions will also be accepted. The software can be written in any programming language, as long as the compiler/interpreter is free to use for academic purposes. One exception is MATLAB, which will be accepted (version 2016b or newer) since it is popular in the graphics community. If you choose to use MATLAB, you may use any standard or custom toolbox and you may have mex functions, as long as they satisfy the license requirements described below. Detailed instructions to install the compilers/interpreter and the dependencies should be available in the readme file.

2. Replicating Results

A script should be provided to reproduce one representative figure or table in the paper that contains a result generated with the proposed algorithm. The script should run without parameters and generate the data that is shown in the figure or listed in the table. If the produced data is not in a standard format, please describe the format in a text file attached to the submission. The script may either directly render the image or generate the data that has been used to produce the image. The review process will only ensure that the code runs and is able to reproduce the material for the figure. The quality and robustness of the code will not be evaluated and they will have no impact on the review process.

3. Interactive Applications

If your application is interactive and it is not possible to script it, you must provide a short movie clip that captures the screen while you use your software to reproduce the representative figure. The reviewers will then follow step by step the video to reproduce the same result. If you want to use this option, you will have to provide a short description to justify why it is not possible to script the generation of the results.

4. Data Restrictions

There might be cases where the data used in a paper cannot be publicly released with the same license as the code. For example, it is common to have additional clauses that a user needs to accept before being allowed to download the data. In these cases, it is sufficient to provide a url that points to a website that allows users to apply for (or buy) the license needed to obtain the data. A confidential copy of the data (which will be deleted after the review) should be sent to the reviewer.

5. Dataset Papers

The main contribution of a dataset paper is the acquisition or creation of a large dataset, which is useful for the community. In these cases, the stamp will be awarded if the authors released their dataset freely for non-commercial usage — it is not mandatory (but still encouraged) for the authors to release the code to replicate (and/or regenerate if possible and appropriate) the entire dataset or the figures in the accompanying paper.

6. User Studies

For some papers the main contribution is reporting about data/measure collections created out of user studies. These papers typically reach useful conclusions based on a statistical analysis over such data. In these cases, the stamp will be awarded if the authors released the data collected freely for non-commercial usage, along with a script to process this data and replicate one of the figures/tables shown in the paper. Note that if the paper also includes an algorithmic contribution, then the code of such an algorithm is required too.

7. Long Running Times

The authors should provide precomputed data for each result/algorithmic step requiring more than 24 hours. The code and training data required to reproduce the result/algorithmic step should still be in the repository, but the reviewer will not rerun the computation to save time and resources. This applies in particular to training of deep networks or expensive parameter searches: for these cases, providing the weights of the network or the parameters will allow the reviewer to validate the results, without having to recompute them. While highly discouraged, it is possible to opt out of releasing the data if there are copyright problems as explained in (4).

8. Hardware Requirements

We encourage the authors to provide code that runs without requiring specific hardware, to favor portability and simplify the replicability of the results. However, if this is not possible, we will accept submissions that run on specific hardware as long as this is justified in the submission. The following non-standard hardware is currently allowed: NVIDIA Tesla, NVIDIA GTX 1080, NVIDIA TITAN BLACK, and AMD Firepro D700. If the necessary hardware is not included on this list, please contact the Replicability Committee Chair.

9. Custom Hardware

Submissions containing a custom hardware component (such as a new custom display or VR technology) can also apply for the stamp, as long as the submission also contains a software contribution. The replicability stamp review process will validate only the software part. In these cases, the authors should provide two additional archives in the code repository: (1) the complete hardware blueprints (which must contain sufficient information to replicate the hardware), and (2) the raw data acquired with the system and that it is used to create the figures/tables in the submission.

10. License

The code must be distributed with a license that allows free non-commercial usage. Its dependencies must have a license that allows free non-commercial or academic usage. Note that this does not preclude charging your users for commercial usage (the CGAL project is a successful example of this code distribution model). For example, you can have Mosek or Gurobi as a dependency, since they are free for academic purposes, while KNitro cannot be used since they do not provide a free non-commercial or academic license. The only exception to this rule is the use of MATLAB, due to its popularity in our community. If your project cannot satisfy this rule, please contact the chair to discuss the special case.

11. Exceptions

The previous rules are general guidelines: If your software cannot satisfy one of the previous requirements, please write as soon as possible to the Replicability Committee Chair to explain the reasons. In many cases, it will be possible to lift any of the previous requirements if there is a good reason for doing so.

12. Submission Procedure

The submission for the stamp takes place only after the paper is accepted at a journal or conference participating in the GRSI. The review process is not blind: the authors and reviewers are aware of each other's identities. A single reviewer will be assigned to each submission and will work with the authors to ensure that the submission is improved until it satisfies all the guidelines. The authors must follow the specific instructions published on the website of the corresponding journal. The repository should contain:
  • A txt file containing the title of the submission, the authors, and the operating system.
  • A script that automatically downloads all required dependencies and compiles the code. If this is not possible (for example if MATLAB needs to be installed), then a text file with precise instructions should be provided. The instructions should describe all the software dependencies that need to be installed, together with detailed compilation instructions. Please always specify entire paths: for example, do not write "change the cmake variable sx to point to the directory where solver x is installed" but instead tell the user to install the solver x in "c:/solverx" and always use absolute paths (assuming a vanilla installation of the operating system).
  • The source code.
  • A script for one representative figure or table created with the proposed algorithm, which runs the provided implementation and generates the data used to create the figure. In case of a comparison figure, the script only needs to reproduce the parts created with the proposed algorithm. Note that the script does not necessarily need to create a figure: if the figure shows a triangle mesh, it is sufficient to export the triangle mesh into a text file.
  • A liability form that gives permission to the reproducibility committee and reviewers to review the code and advertise the review publicly after the stamp is approved.