Submission Requirements

To ensure a fair and efficient review of the code submissions, the following guidelines must be followed when you prepare your submission. If there are technical problems that prevent you from complying with the requirements, please contact the Chair of the Replicability Committee.

1. Distribution

The software must compile and run on \emph{at least one} of the following operating systems, using exactly one of the versions listed below:
  • Windows 10 or newer
  • Ubuntu 16.04.1 LTS or newer
  • MacOSX 10.12 or newer
Note that you can pick whichever operating system you developed your code on. Your code does not need to compile on all of them to obtain the stamp. The code must compile and run on a vanilla installation, without any software installed. If you need specific software to be installed, you should provide a script that automatically downloads and installs all the required dependencies. In the case of Windows, a text file with precise installation instructions will also be accepted. The software can be written in any programming language, as long as the compiler/interpreter is free to use for academic purposes. One exception is MATLAB, which will be accepted (version 2016b or newer) since it is popular in the graphics community. If you choose to use MATLAB, you may use any standard or custom toolbox and you may have mex functions, as long as they satisfy the license requirements described below. Detailed instructions to install the compilers/interpreter and the dependencies should be available in the readme file.

2. Replicating Results

An independent script should be provided for every figure or table of the paper that contains a result generated with the algorithm. The script should run without parameters and generate the data that is shown in the figure or listed in the table. If the produced data is not in a standard format, please describe the format in a text file attached to the submission. The script may either directly render the image or generate the data that has been used to produce the image. The review process will only ensure that the code runs and is able to reproduce the material for the figures. The quality and robustness of the code will not be evaluated and they will have no impact on the review process.

3. Timings

If the paper includes timings, an additional script should be included in the submission. The script should run all the experiments for which a timing is provided in the paper, and save the results in a single text file. The reviewer will launch the scripts on their machine --- the produced txt file will be attached as additional material to the submission, together with a description of the CPU, GPU and maximal memory of the workstation used to replicate the experiments. Note that it is not expected that the timings in the paper will match those obtained by the reviewer.

4. Interactive Applications

If your application is interactive and it is not possible to script it to replicate a certain figure in the paper, you must provide a short movie clip that captures the screen while you use your software to reproduce the figure. The reviewers will then follow step by step the video to reproduce the same result. If you want to use this option, you will have to provide a short description to justify why it is not possible to script the generation of the results.

5. Data Restrictions

There might be cases where the data used in a paper cannot be publicly released with the same license as the code. For example, it is common to have additional clauses that a user needs to accept before being allowed to download the data. In these cases, it is sufficient to provide a url that points to a website that allows users to apply for (or buy) the license needed to obtain the data. A confidential copy of the data (which will be deleted after the review) should be sent to the reviewer.

6. Dataset Papers

The main contribution of a dataset paper is the acquisition or creation of a large dataset, which is useful for the community. In these cases, the stamp will be awarded if the authors released their dataset freely for non-commercial usage — it is not mandatory (but still encouraged) for the authors to release the code to replicate (and/or regenerate if possible and appropriate) the entire dataset or the figures in the accompanying paper.

7. Hardware Requirements

We encourage the authors to provide code that runs without requiring specific hardware, to favor portability and simplify the replicability of the results. However, if this is not possible, we will accept submissions that run on specific hardware as long as this is justified in the submission. The following non-standard hardware is currently allowed: NVIDIA Tesla, NVIDIA GTX 1080, NVIDIA TITAN BLACK, and AMD Firepro D700. If the necessary hardware is not included on this list, please contact the Replicability Committee Chair.

8. License

The code must be distributed with a license that allows free non-commercial usage. Its dependencies must have a license that allows free non-commercial or academic usage. Note that this does not preclude charging your users for commercial usage (the CGAL project is a successful example of this code distribution model). For example, you can have Mosek or Gurobi as a dependency, since they are free for academic purposes, while KNitro cannot be used since they do not provide a free non-commercial or academic license. The only exception to this rule is the use of MATLAB, due to its popularity in our community. If your project cannot satisfy this rule, please contact the chair to discuss the special case.

9. Exceptions

The previous rules are general guidelines: If your software cannot satisfy one of the previous requirements, please write as soon as possible to the Replicability Committee Chair to explain the reasons. In many cases, it will be possible to lift any of the previous requirements if there is a good reason for doing so.

10. Submission Procedure

The submission for the stamp takes place only after the paper is accepted at a journal or conference participating in the GRSI. The review process is not blind: the authors and reviewers are aware of each other's identities. A single reviewer will be assigned to each submission and will work with the authors to ensure that the submission is improved until it satisfies all the guidelines. The authors must follow the specific instructions published on the website of the corresponding journal. The repository should contain:
  • A txt file containing the title of the submission, the authors, and the operating system.
  • A script that automatically downloads all required dependencies and compiles the code. If this is not possible (for example if MATLAB needs to be installed), then a text file with precise instructions should be provided. The instructions should describe all the software dependencies that need to be installed, together with detailed compilation instructions. Please always specify entire paths: for example, do not write "change the cmake variable sx to point to the directory where solver x is installed" but instead tell the user to install the solver x in "c:/solverx" and always use absolute paths (assuming a vanilla installation of the operating system).
  • The source code.
  • A script for every result figure created with the proposed algorithm, which runs the provided implementation and generates the data used to create the figure. In case of a comparison figure, the script only needs to reproduce the parts created with the proposed algorithm. Note that the script does not necessarily need to create a figure: if the figure shows a triangle mesh, it is sufficient to export the triangle mesh into a text file.
  • A script that produces a text file with all the timings. (This is only needed if you report timings in the submission).
  • A liability form that gives permission to the reproducibility committee and reviewers to review the code and advertise the review publicly after the stamp is approved.